Improve time to join a remote room

[kittykat]

Users expect joining a remote room to be fast - a few seconds at most.

The current work being undertaken is protocol level work to allow you to "partially" join a remote room, which allows you to send and receive events, while the join fully completes in the background. End of Q3 we put it out for community testing, with the caveats it a) has lots of edge cases and bugs, and b) doesn't work for worker deployments.

The backend team are currently estimating the remaining work before it can be enabled by default across all deployments.

Note: there is a separate (smaller) tranche of work to actually optimise the code involved in the presence of faster remote room joins. So once the above is complete there might still be a bunch of work to do to get it to a few seconds at most.

Relevant milestones:

• Q3 milestones
• Q4 milestones

These tasks need to be completed for this issue to be closed:

• Faster joins: improve handling of race where we can fail to persist a client event with partial state after _sync_partial_state_room clears the partial state flag for a room #13173
• Faster joins: non-lazy_load_members /syncs should not block #12989
• Faster joins: send heroes to handle nameless rooms #12995
• Faster joins: support sending messages from clients during the half-joined state #12997
• Faster joins: Handle leaving a room while re-syncing state #12802
• Faster joins: Handle a second join while syncing state for the first one #12801
• Faster joins: fixes to device list tracking #12993
• Faster joins: Update room stats once state re-sync completes #12814
• Faster joins: Update user directory once state re-sync completes #12815
• Faster joins: support worker-mode deployments #12994
• Faster joins: check compatibility with history import #13003
• Faster joins: room alias server list may be incomplete #13287
• Faster joins: smarter algorithm for picking a server to resync from #12999
• Complement TestPartialStateJoin/Resync_completes_even_when_events_arrive_before_their_prev_events is flaky #13777
• Complement TestPartialStateJoin/Rejects_send_knock_during_partial_join is flakey(?) #13945
• Complement flake(?) TestPartialStateJoin/* due to complement hostname:port re-use #13975

[richvdh]

Related (possibly a duplicate of?): #1211

[erikjohnston]

This is now live, so I think we should close this tracking issue.