Add developer documentation to explain room DAG concepts like outliers and state_groups

[MadLittleMods]

Add developer FAQ to explain room DAG concepts like outliers and state_groups

• outlier explanation pulled from my understanding working on MSC2716 and various comments throughout the code:
  • synapse/handlers/federation.py#L2196-L2199
  • synapse/handlers/federation.py#L2731-L2735
  • synapse/federation/federation_client.py#L259-L261
• state_group explanation pulled from Make historical events discoverable from backfill for servers without any scrollback history (MSC2716) (federation) #10245 (comment) by @erikjohnston
• Depth and stream ordering from the following comments:
  • @erikjohnston, Add endpoints for backfilling history (MSC2716) #9247 (comment)
  • @richvdh, Synapse returned event in backfill that was already returned during sync #7164 (comment)
  • @richvdh, Synapse returned event in backfill that was already returned during sync #7164 (comment)

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
• Pull request includes a sign off
• Code style is correct (run the linters)

[dklimpel]

Also the summary needs an update / entry.
https://github.com/matrix-org/synapse/blob/develop/docs/SUMMARY.md

[MadLittleMods]

Thanks for the shout @dklimpel! Added 👍

[richvdh]

thanks very much for this, @MadLittleMods - we're definitely lacking some useful documentation in this area!

[richvdh]

I think it's slightly different. The thing that makes an outlier an outlier is that we haven't figured out the state for the room at that point in the DAG.

We won't necessarily have its prev_events in the database, but it's entirely possible that we will. Conversely, it's possible for us not to have all the prev_events for non-outlier event (in which case, that event would be a "backward extremity").

The process of transforming an outlier to a non-outlier involves figuring out the room state at that point in the DAG - either by having the state at all the prev_events and resolving the state between them, or simply by asking another server.

auth events are a whole extra level of complexity, but even for an outlier we should have all the events in the auth chain.

synapse/synapse/handlers/federation.py

Lines 2196 to 2199 in cd5fcd2

	Normally, our calculated auth_events based on the state of the room
	at the event's position in the DAG, though occasionally (eg if the
	event is an outlier), may be the auth events claimed by the remote
	server.

is quite a specific bit of code relating to one part of processing incoming events (and to be honest, I'm not convinced it does the right thing). I would leave auth events out of this explanation, tbh.

[MadLittleMods]

Thanks for all info @richvdh 😀

These seem so closely coupled and aren't mutually exclusive in real life situations so it's hard for me to make a clear description 🤯

I've updated with something more inline with your description but feel free to rewrite.

I've mainly been working with floating outliers so I don't think I had the complete picture.

---

I feel like I don't understand where an outlier becomes an ex_outlier.

Say we fetch a missing auth event for an event, we mark that auth event as an outlier. Then at what point later, do we get the event again as a non-outlier? From your statement, it's not when we have fetched all of the prev_events as those could already be all persisted in the db.

There is _update_outliers_txn which handles ex_outlier stuff but it's confusing where/when we will come across the same outlier event again while persisting.

Based on the DAG below, as a remote federating server, my assumption is that we process event D which fetches the state_event as an outlier. Then as we backfill more, and process state_event directly, it becomes an ex_outlier.

Mermaid live editor playground link

I'm still a bit wary on a situation where we could have all of the prev_events for the state_event but still mark it as an outlier. Perhaps with gaps in the DAG and the event hits it just right 🤷

[richvdh]

I've mainly been working with floating outliers so I don't think I had the complete picture.

I think I might have misled you here. From Synapse's point of view, all outliers are considered to be floating, and in the majority of cases we indeed won't have their prev events. However, it's possible for the stars to align in a way that we do actually have their prev events in the database - at least as outliers themselves.

To take a simple example:

Two of the auth events for a join event in an invite-only room are the join_rules event and an invite event for the joiner. So let's suppose we fetch both of those events as outliers. Let's also suppose that the invite was issued immediately after changing the join rules - so the join_rules is the only prev-event of the invite event.

join_rules <-  invite
   ^             ^
   ............................. join

In that case, it so happens that we have all the prev-events of the invite event - but we're still considering it an outlier.

So I don't want you to imagine there is a distinction between "floating outliers" and "regular outliers" - they are all just outliers in terms of how we handle them in Synapse.

Say we fetch a missing auth event for an event, we mark that auth event as an outlier. Then at what point later, do we get the event again as a non-outlier? From your statement, it's not when we have fetched all of the prev_events as those could already be all persisted in the db.

right. It becomes a non-outlier in the situation where we process it as a regular event as part of the timeline - normally via backfill.

Based on the DAG below, as a remote federating server, my assumption is that we process event D which fetches the state_event as an outlier. Then as we backfill more, and process state_event directly, it becomes an ex_outlier.

exactly so, yes.

I'm still a bit wary on a situation where we could have all of the prev_events for the state_event but still mark it as an outlier. Perhaps with gaps in the DAG and the event hits it just right 🤷

yup. Imagine there is another fork in the DAG pointing to A:

When we receive Y, we'll backfill A. So then we have state_event's prev_events - but state_event is still an outlier at this point.

[MadLittleMods]

Thanks for the clarification here. I've added a paragraph explaining that there is no distinction.

I like these examples and edge cases. I'm tempted to add them into the docs but I think I'll save that for another iteration.

[MadLittleMods]

Thanks for all of the review so far @richvdh. My mind is being blown in multiple ways! 🤯

[richvdh]

I think this is a bit confusing. How about:

Suggested change

	extremity and those `prev_events` become the new backwards extremities (unless
	we have already persisted them). Also in reality, we backfill in batches of 20
	events or so, so only the `prev_events` of the last oldest-in-time event will
	become the backwards extremeties.
	extremity (although we may have formed new backwards extremities during the backfilling process).

[MadLittleMods]

👍 I tweaked it slightly to include the prev event chain,

Once we have fetched all of its prev_events, it's unmarked as a backwards
extremity (although we may have formed new backwards extremities from the prev
events during the backfilling process).

[richvdh]

lgtm!

[richvdh]

thanks for your perseverence, @MadLittleMods!

[MadLittleMods]

Thank you for the meaty review @richvdh to get this factually correct! Lots of great tidbits shared 🦌