diff --git a/docs/src/developer/developer/large-conversations.md b/docs/src/developer/developer/large-conversations.md new file mode 100644 index 0000000000..bda4349e74 --- /dev/null +++ b/docs/src/developer/developer/large-conversations.md @@ -0,0 +1,69 @@ +# Refactoring galley to support large conversations + +To be able to suppport large conversations galley needs refactoring. This +section in the developer docs is meant to contain useful references for this +endeavor. + +## Call hierarchy + +See [large-conversations.yaml](https://github.com/wireapp/wire-server/blob/develop/docs/src/developer/developer/large-conversations.yaml) for + call hierarchy of functions that load the full member list into memory. + +## Galley Endpoints + +These are all the endpoints in galley with response types that contain the full member list of conversations. + +In the `ConversationAPI`: + +For type `Conversation`: +- `get-unqualified-conversation` +- `get-unqualified-conversation-legalhold-alias` +- `get-conversation@v2` +- `get-conversation` +- `get-mls-self-conversation` + +For type `ConversationList Conversation` +- `get-conversations` + +For type `ConversationsResponse`: +- `list-conversations@v1` +- `list-conversations@v2` +- `list-conversations` + +For type `ConversationResponse`: +- `create-group-conversation@v2` +- `create-group-conversation@v3` +- `create-one-to-one-conversation@v2` +- `create-one-to-one-conversation` + +For type `CreateGroupConversationResponse`: +- `create-group-conversation` +- `create-self-conversation@v2` +- `create-self-conversation@v2` +- `create-self-conversation` + +For type `Wire.API.Event.Conversation.Event` (`EventData` might contain a `Conversation` object, but it's not clear from API type alone if it contains one) +- `add-members-to-conversation-unqualified` +- `add-members-to-conversation-unqualified2` +- `add-members-to-conversation` +- `join-conversation-by-id-unqualified` +- `join-conversation-by-code-unqualified` +- `create-conversation-code-unqualified` +- `remove-code-unqualified` +- `remove-member-unqualified` +- `update-conversation-name-deprecated` +- `update-conversation-name-unqualified` +- `update-conversation-name` +- `update-conversation-message-timer-unqualified` +- `update-conversation-message-timer` +- `update-conversation-receipt-mode` +- `update-conversation-access-unqualified` +- `update-conversation-access@v2` +- `update-conversation-access` + +In the `MLSAPI`: + +- `mls-commit-bundle` returns a `MLSMessageSendingStatus` which contains list of `Event`, as well a list of unreachable users. +- `mls-message-v1` and `mls-message` returns list of `Event` + +The `MessagingAPI` (Proetus) is not listed here, assuming that Proteus conversations won't support large conversations. diff --git a/docs/src/developer/developer/large-conversations.yaml b/docs/src/developer/developer/large-conversations.yaml new file mode 100644 index 0000000000..4f39885162 --- /dev/null +++ b/docs/src/developer/developer/large-conversations.yaml @@ -0,0 +1,797 @@ +# Context: We would like to refactor galley so the full list of conversation +# members is never loaded into memory. This refactoring a requiste for +# supporting large conversations (10k, 100k users). +# +# This file is the outcome of mapping the call hierarchy of these functions in +# galley: +# +# Galley.Cassandra.Conversation.Members.members +# Galley.Cassandra.Conversation.Members.lookupMLSClients +# Galley.Cassandra.Conversation.Members.lookupRemoteMembers +# +# These functions all fetch the full member list of a conversation. The call +# hierarchy is encoded in the "dependents" field, e.g. +# +# - name: members +# dependents: +# - Galley.Effects.ConversationStore.getConversation +# +# means that `Galley.Effects.ConversationStore.getConversation` calls `members`. +# You can think of the call hierarchy as a directed (acyclic) graph with +# `members`, `lookupMLSClients`, `lookupRemoteMembers` at the top. The leaves of +# the graph are functions which are API handlers or "main" function of +# background threads: they are not called by any other functions. +# +# For each function in the graph I judged if it can be refactored so that it +# doesn't load the full member list into memory. I also judged if that change +# can be "compatible", i.e. all its dependents can continue using the function +# without any interface / behaviour change or whether the refactoring is +# "breaking" the API contract with its dependents / endpoints. If the +# refactoring needs helper functions that don't exist yet I've added them in the +# "new_things" section at the end of this file. Any new helper functions which +# are needed I've listed in the "change_needs" field. +# +# I hope this file is useful in making a refactoring plan, estimating effort and +# keeping track of the refactoring progress. It probably is easiest to start +# refactoring at the leaves, gradually refactoring up the call hierarchy until +# eventually the member-loading functions are not used anymore and can be +# deleted. +# + +functions: + - name: Galley.Cassandra.Conversation.Members.members + change: breaking + comments: | + Delete it! + dependents: + - Galley.Effects.ConversationStore.getConversation + - Galley.Cassandra.Conversation.deleteConversation + + - name: Galley.Cassandra.Conversation.Members.lookupMLSClients + change: breaking + comments: | + Delete it! + dependents: + - Galley.Effects.MemberStore.lookupMLSClients + + - name: Galley.Cassandra.Conversation.Members.lookupRemoteMembers + change: breaking + comments: | + delete it! + dependents: + - Galley.Cassandra.Conversation.deleteConversation + - Galley.Cassandra.Conversation.getConversation + - Galley.Cassandra.Conversation.localConversation + - Galley.Effects.MemberStore.getRemoteMembers + + - name: Galley.Effects.MemberStore.lookupMLSClients + change: breaking + comments: | + delete it! + dependents: + - Galley.API.MLS.Message.postMLSCommitBundleToLocalConv + - Galley.API.MLS.Message.postMLSMessageToLocalConv + - Galley.API.MLS.Removal.removeClient + - Galley.API.MLS.Removal.removeUser + + - name: Galley.API.Clients.rmClientH + change: compatible + comments: | + member list use by removeClientsWithClientMap + to propagate backend remove proposal for client that is leaving conv + change_needs: + - removeClientsWithClientMap' + + - name: Galley.API.Create.createConnectConversation + change: breaking + comments: | + breaking because of PublicConversationViewWithoutMembers + change_needs: + - getConversation' + - createConversation' + - newPushLocal' + - push1' + - notifyCreatedConversation' + - isMemberOfLocalConv + - hasLocalConvMembers + + - name: Galley.API.Create.createGroupConversationGeneric + change: breaking + dependents: + - Galley.API.Create.createGroupConversationUpToV3 + - Galley.API.Create.createGroupConversation + comments: | + wouldnt be able to get (Set (Remote User)) of failedToNotify users + change_needs: + - createConversation' + - getConversation' + + - name: Galley.API.Create.createGroupConversationUpToV3 + change: breaking + comments: | + would respond with ResponseForExistedCreated PublicConversation' + + - name: Galley.API.Create.createGroupConversation + change: breaking + comments: | + would respond with ResponseForExistedCreated CreateGroupConversationResponse' + + - name: Galley.API.Federation.onClientRemoved + change: compatible + comments: | + See plan for Galley.API.Clients.rmClientH + + - name: Galley.API.Federation.onUserDeleted + change: compatible + change_needs: + - isRemoteMember' + - notifyConversationAction' + + - name: Galley.API.MLS.Util.getLocalConvForUser + change: breaking + comments: | + returns Conversation' + dependents: + - Galley.API.MLS.GroupInfo.getGroupInfoFromLocalConv + - Galley.API.MLS.Message.postMLSCommitBundleToLocalConv + - Galley.API.MLS.Message.postMLSMessageToLocalConv + - Galley.API.MLS.Message.processExternalCommit + + - name: Galley.API.MLS.GroupInfo.getGroupInfoFromLocalConv + change: compatible + change_needs: + - isMember' + + - name: Galley.API.MLS.Message.postMLSCommitBundleToLocalConv + chanage: breaking + comments: | + Maybe compatible, depends on unreachables, do we need them? + change_needs: + - isMember' + - getConversation' + - propagateMessage' + + - name: Galley.API.MLS.Message.postMLSMessageToLocalConv + chanage: breaking + comments: | + Maybe compatible, depends on unreachables, do we need them? + change_needs: + - isMember' + - getConversation' + + - name: Galley.API.MLS.Message.processExternalCommit + change: compatible + comments: | + Argument would change from Conv to ConvId + change_needs: + - removeClientsWithClientMap' + + - name: Galley.API.LegalHold.handleGroupConvPolicyConflicts + change: compatible + comments: | + This functions need to be kept as blocking. + change_needs: + - getConvLocalMembersPage + + - name: Galley.API.Message.postQualifiedOtrMessage + change: ??? + comments: | + Proteus. Not sure if we want to do this. + + - name: Galley.API.One2One.iUpsertOne2OneConversation + change: compatible + change_needs: + - getConversation' + - hasLocalConvMembers + - hasRemoteConvMembers + + - name: Galley.API.Query.getConversationByReusableCode + change: compatible + change_needs: + - ensureConversationAccess' + + - name: Galley.API.Query.getConversationGuestLinksStatus + change: compatible + change_needs: + - getConversation' + - ensureConvAdmin' + + - name: Galley.API.Query.getLocalSelf + change: compatible + comments: | + Includes weird cleanup code that deletes the conversation if it is not "alive" + change_needs: + - getLocalMember' + + - name: Galley.API.Query.getMLSSelfConversation + change: breaking + dependents: + - Galley.API.Query.getMLSSelfConversationWithError + comments: | + Maybe optional? + returns PublicConversation' + change_needs: + - getConversation' + + - name: Galley.API.Query.getMLSSelfConversationWithError + change: ??? + comments: | + Maybe optional? + + - name: Galley.API.Teams.uncheckedDeleteTeamMember + change: compatible + dependents: + - newPushLocal' + - push1' + + - name: Galley.API.Create.createLegacyOne2OneConversationUnchecked + change: compatible + change_needs: + - getConversation' + + - name: Galley.API.Create.createOne2OneConversationLocally + change: compatible + change_needs: + - getConversation' + + - name: Galley.API.Create.createProteusSelfConversation + change: ??? + change_needs: + - getConversation' + + - name: Galley.API.Update.acceptConv + change: compatible + change_needs: + - getConversation' + + - name: Galley.API.Update.addBot + change: compatible + change_needs: + - isMember' + - getLocalMember' + - ensureConversationAccess' + - ensureMemberLimit' + + - name: Galley.API.Update.addCode + change: compatible + change_needs: + - getConversation' + - ensureConvAdmin' + - pushConversationEvent' + + - name: Galley.API.Update.blockConv + change: compatible + change_needs: + - getConversation' + - isMember' + + - name: Galley.API.Update.checkReusableCode + change: compatible + change_needs: + - getConversation' + + - name: Galley.API.Update.getCode + change: compatible + change_needs: + - getConversation' + - isMember' + + - name: Galley.API.Update.joinConversationById + change: breaking + change_needs: + - getConversation' + + - name: Galley.API.Update.rmBot + change: compatible + change_needs: + - getConversation' + - isMember' + - getLocalMember' + - ensureActionAllowed' + - push1' + + - name: Galley.API.Update.rmCode + change: compatible + change_needs: + - getConversation' + - ensureConvAdmin' + - ensureConversationAccess' + - pushConversationEvent' + + - name: Galley.API.Update.unblockConv + change: breaking + change_needs: + - getConversation' + + - name: Galley.API.Update.joinConversation + change: breaking + comments: | + This is a big one + dependents: + - Galley.API.Update.joinConversationById + - Galley.API.Update.joinConversationByReusableCode + change_needs: + - notifyConversationAction' + + - name: Galley.API.Update.joinConversationByReusableCode + change: breaking + change_needs: + - getConversation' + + - name: Galley.API.Util.getConversationWithError + change: breaking + dependents: + - Galley.API.Util.getConversationAndMemberWithError + - Galley.API.Util.updateLocalConversation + change_needs: + - getConversation' + + - name: Galley.API.Util.getConversationAndMemberWithError + change: breaking + comments: | + return (Conversation', mem) + dependents: + - Galley.API.Query.getBotConversation + - Galley.API.Util.getConversationAndCheckMembership + - Galley.API.Federation.leaveConversation + - Galley.API.Update.memberTyping + - Galley.API.Federation.updateTypingIndicator + + - name: Galley.API.Util.getConversationAndCheckMembership + change: breaking + dependents: + - Galley.API.Query.getUnqualifiedConversation + - Galley.API.Query.getConversationRoles + + - name: Galley.API.Query.getUnqualifiedConversation + change: breaking + dependents: + - Galley.API.Query.getConversation + + - name: Galley.API.Query.getConversation + change: breaking + + - name: Galley.API.Query.getConversationRoles + change: compatible + change_needs: + - getConversation' + - isMember' + + - name: Galley.API.Query.getBotConversation + change: compatible + change_needs: + - isMember' + + - name: Galley.API.Federation.leaveConversation + change: compatible + change_needs: + - getConversation' + - isMember' + - notifyConversationAction' + + - name: Galley.API.Update.memberTyping + change: compatible + change_needs: + - push1' + - isRemoteMember' + + - name: Galley.API.Federation.updateTypingIndicator + change: compatible + change_needs: + - push1' + + - name: Galley.Cassandra.Conversation.localConversation + change: breaking + dependents: + - Galley.Effects.ConversationStore.getConversations + + - name: Galley.Effects.ConversationStore.getConversations + change: breaking + dependents: + - Galley.API.Federation.getConversations + - Galley.API.Internal.rmUser + - Galley.API.Query.getConversationsInternal + - Galley.API.Query.listConversations + + - name: Galley.API.Federation.getConversations + change: breaking + dependents: + - fed enpoint "get-conversations" + + - name: Galley.API.Internal.rmUser + change: compatible + dependents: + change_needs: + - getConversation' + + - name: Galley.API.Query.getConversationsInternal + change: breaking + dependents: + - Galley.API.Query.getConversations + - Galley.API.Query.iterateConversations + + - name: Galley.API.Query.getConversations + change: breaking + + - name: Galley.API.Query.iterateConversations + change: breaking + dependents: + - Galley.API.LegalHold.handleGroupConvPolicyConflicts + + - name: Galley.API.Query.listConversations + change: breaking + + - name: Galley.API.Action.performAction + change: breaking + comments: | + centeral one + dependents: + - Galley.API.Action.updateLocalConversationUnchecked + change_needs: + - ensureMemberLimit' + - ensureConversationAccess' + - performConversationJoin' + - getLocalMember' + - getRemoteMember' + + - name: Galley.API.Action.updateLocalConversationUnchecked + change: breaking + dependents: + - Galley.API.Action.updateLocalConversation + - Galley.API.MLS.Message.executeProposalAction + change_needs: + - Galley.API.Action.performAction' + - notifyConversationAction' + + - name: Galley.API.Action.updateLocalConversation + change: breaking + dependents: + - Galley.API.Action.updateLocalConversationUnchecked' + + - name: Galley.API.MLS.Message.executeProposalAction + change: breaking + comments: | + because ConversationUpdate contains cuAlreadyPresentUsers (federation api) + change_needs: + - lookupMLSClients' + + - name: Galley.API.Federation.updateConversation + change: breaking + comments: | + no more cuAlreadyPresentUsers + + - name: Galley.API.Internal.deleteLoop + change: compatible + + - name: Galley.API.Teams.deleteTeamConversation + change: compatible + change_needs: + - Galley.API.Action.updateLocalConversation + + - name: Galley.API.Teams.uncheckedDeleteTeam + change: compatible + dependents: + - Galley.API.Internal.deleteLoop + comments: | + This loops over all team members, not conversation members. + So it's still problematic. Should do this in pages to avoid OOM. + + - name: Galley.API.Update.deleteLocalConversation + change: compatible + comments: | + The event is probably a conversation delete event not containting any members + dependents: + - Galley.API.Teams.deleteTeamConversation + + - name: Galley.Cassandra.Conversation.getConversation + change: breaking + dependents: + - Galley.Effects.ConversationStore.getConversation + + - name: Galley.Cassandra.Team.deleteTeam + change: compatible + dependents: + - Galley.Effects.TeamStore.deleteTeam + + - name: Galley.Effects.TeamStore.deleteTeam + change: compatible + dependents: + - Galley.API.Teams.uncheckedDeleteTeam + + - name: Galley.Effects.TeamStore.deleteTeamConversation + change: compatible + dependents: + - Galley.API.Action.updateLocalConversationUnchecked' + + - name: Galley.Cassandra.Conversation.deleteConversation + change: compatible + dependents: + - Galley.Cassandra.Team.removeTeamConv + - Galley.Effects.ConversationStore.deleteConversation + + - name: Galley.Cassandra.Team.removeTeamConv + change: compatible + dependents: + - Galley.Cassandra.Team.deleteTeam + - Galley.Effects.TeamStore.deleteTeamConversation + + - name: Galley.Effects.ConversationStore.deleteConversation + change: compatible + dependents: + - Galley.API.Action.performAction + - Galley.API.Query.getConversationsInternal + - Galley.API.Query.listConversations + - Galley.API.Query.getLocalSelf + + - name: Galley.Effects.ConversationStore.getConversation + change: breaking + comments: | + Remove the member list + dependents: + - Galley.API.Clients.rmClientH + - Galley.API.Create.createGroupConversationGeneric + - Galley.API.Create.createProteusSelfConversation + - Galley.API.Create.createLegacyOne2OneConversationUnchecked + - Galley.API.Create.createOne2OneConversationLocally + - Galley.API.Create.createConnectConversation + - Galley.API.Federation.onClientRemoved + - Galley.API.Federation.onUserDeleted + - Galley.API.MLS.Util.getLocalConvForUser + - Galley.API.Message.postQualifiedOtrMessage + - Galley.API.One2One.iUpsertOne2OneConversation + - Galley.API.Query.getConversationByReusableCode + - Galley.API.Query.getConversationGuestLinksStatus + - Galley.API.Query.getMLSSelfConversation + - Galley.API.Teams.uncheckedDeleteTeamMember + - Galley.API.Update.acceptConv + - Galley.API.Update.blockConv + - Galley.API.Update.unblockConv + - Galley.API.Update.addCode + - Galley.API.Update.rmCode + - Galley.API.Update.getCode + - Galley.API.Update.checkReusableCode + - Galley.API.Update.joinConversationByReusableCode + - Galley.API.Update.joinConversationById + - Galley.API.Update.addBot + - Galley.API.Update.rmBot + - Galley.API.Util.getConversationWithError + + - name: Galley.Cassandra.Conversation.Members.addMembers + change: breaking + comments: | + we should restrict the number of users you can add in the dependents + dependents: + - Galley.Cassandra.Conversation.createMLSSelfConversation + - Galley.Cassandra.Conversation.createConversation + - Galley.Effects.MemberStore.createMember + + - name: Galley.Cassandra.Conversation.createMLSSelfConversation + change: breaking + comments: | + Remove member list + + - name: Galley.Cassandra.Conversation.createConversation + change: breaking + comments: | + Remove member list + + - name: Galley.Effects.MemberStore.createMember + change: breaking + comments: | + Remove member list + dependents: + - Galley.Effects.ConversationStore.createConversation + + - name: Galley.Effects.ConversationStore.createConversation + change: breaking + comments: | + Remove member list + dependents: + - Galley.API.Create.createGroupConversationGeneric + - Galley.API.Create.createProteusSelfConversation + - Galley.API.Create.createLegacyOne2OneConversationUnchecked + - Galley.API.Create.createOne2OneConversationLocally + - Galley.API.Create.createConnectConversation + - Galley.API.One2One.iUpsertOne2OneConversation + + - name: Galley.API.Create.createGroupConversationGeneric + change: breaking + comments: | + Remove member list + dependents: + - Galley.API.Create.createGroupConversationUpToV3 + - Galley.API.Create.createGroupConversation + + - name: Galley.API.Create.createGroupConversationUpToV3 + change: breaking + comments: | + Remove member list + + - name: Galley.API.Create.createGroupConversation + change: breaking + comments: | + Remove member list + + - name: Galley.API.Create.createProteusSelfConversation + change: breaking + comments: | + Remove member list + + - name: Galley.API.Create.createLegacyOne2OneConversationUnchecked + change: breaking + comments: | + Remove member list + + - name: Galley.API.Create.createOne2OneConversationLocally + change: breaking + comments: | + Remove member list + + - name: Galley.API.Create.createConnectConversation + change: breaking + comments: | + Remove member list + + - name: Galley.API.One2One.iUpsertOne2OneConversation + change: breaking + comments: | + Remove member list + + - name: Galley.Effects.MemberStore.getLocalMembers + change: breaking + comments: | + Remove member list + dependents: + - Galley.API.Teams.uncheckedDeleteTeam + - Galley.API.Update.updateSelfMember + + - name: Galley.API.Update.updateSelfMember + change: compatible + change_needs: + - isMember' + - isRemoteMember' + + - name: Galley.Effects.MemberStore.getRemoteMembers + change: breaking + comments: | + this is DEAD CODE + + - name: Galley.API.MLS.Removal.removeClient + change: compatible + change_needs: + - removeClientsWithClientMap' + + - name: Galley.API.MLS.Removal.removeUser + change: compatible + change_needs: + - removeClientsWithClientMap' + + +new_things: + + - name: PublicConversation' + description: + public API type that has all the fields of Converation but + without the members list + + - name: Conversation' + description: + internal conversation type that hold all fields except + the member lists + + - name: Push' + description: + Similar to Push but instead of pushRecipients fields the conversation id + exceptions + + - name: getConversation' + description: | + :: ConvId + -> m (Maybe ConversationWithoutMembers) + in Galley.Effects.ConversationStore + + - name: createConversation' + description: | + :: Local ConvId + -> NewConversation + -> ConversationStore m ConversationWithoutMembers + + - name: notifyCreatedConversation' + description: | + :: Local UserId + -> Maybe ConnId + -> ConvId -> + Sem r () + + in Galley.API.Create + + breaking api change. notifyCreatedConversation returned set of remote UserId + that could not be contacted + + - name: newPushLocal' + description: | + :: ListType + -> UserId + -> PushEvent + -> Maybe Push' + in Galley.Intra.Push.Internal + + not clear how to do define this + + - name: push1' + desciption: | + two stratgies (maybe): + strategy 1: consumer loops over pages and calls for each page + this requires that gundeck has fast response times for each call + strategy 2: consumer only gives recipients in form of a convid (+ excpetions) + async gundeck fetches recicpients + + - name: pushConversationEvent' + + - name: removeClientsWithClientMap' + description: | + Local ConvId -> + [KeyPackageRef] -> + Qualified UserId -> + Sem r () + + - name: isMemberOfLocalConv + description: | + :: UserId -> ConvId -> m Bool + + - name: hasLocalConvMembers + description: | + :: ConvId -> m Bool + check if conversation has any members + + - name: hasRemoteConvMembers + description: | + :: ConvId -> m Bool + + - name: isRemoteMember' + description: | + :: Remote UserId -> ConvId -> m Bool + + - name: isMember' + description: | + :: Local UserId -> ConvId -> m Bool + + - name: notifyConversationAction' + description: | + in Galley.API.Action + this one is complicated + + - name: propagateMessage' + description: | + in Galley.API.MLS.Propagate + + - name: getConvLocalMembersPage + + - name: ensureConversationAccess' + + - name: ensureConvAdmin' + + - name: ensureActionAllowed' + + - name: getLocalMember' + description: + in MemberStore + + - name: getRemoteMember' + description: + in MemberStore + + - name: ensureMemberLimit' + decription: | + maybe we don't need this function anymore when large groups + are supported + + - name: performConversationJoin' + description: | + in Galley.API.Action + big and complicated + + - name: lookupMLSClients' + description: | + should replace the lookup in ClientMap + + - name: Galley.API.Action.updateLocalConversationUnchecked'