Plans for Horizontal Scaling

[dyc3]

This is not necessarily complete, and I'll be updating it as I think of new things.

Motivation

OpenTogetherTube is a very complex piece of software, and I want to be prepared to scale up when the time comes. OTT's current architecture is haphazardly planned and pretty inefficient. I've done enough major refactors to know that I want the next one to be the last one. This new architecture needs to be the one that carries the project for years to come.

Constraints

OTT is pretty network efficient, so network throughput is not a huge concern. Modern OSes can handle tons and tons of TCP connections to a single port. However, a more pressing concern is that each user that joins a room puts more load on the Node.js event loop.

I don't have data to back this up yet, so for now it's a hypothesis.

It needs to be possible to spin up and take down monoliths on demand. It also needs to be possible to spin up and take down new balancers easily as well.

Leader Election and why it won't work

Leader Election is a distributed computing concept about getting different nodes in a network to agree on which one of the nodes has control over a resource. Read more here. For OTT, this would mean which node has control over a room's state.

Algorithms that perform leader election are extremely complicated, and from what I've seen, may not even guarentee that exactly 1 node is elected as leader.

Smart Load Balancers

The idea here is to make a special load balancer that knows how loaded each node is, and can assign rooms to nodes.

Topographically, it would look something like this:

graph LR
    bal1{Load Balancer}
    bal1 --> A[Node]
    bal1 --> B[Node]
    bal1 --> C[Node]
    bal2{Load Balancer}
    bal2 --> A[Node]
    bal2 --> B[Node]
    bal2 --> C[Node]

    client[Lots of clients] --> bal1
    client --> bal2

    A --> db[(Redis & Postgres)]
    B --> db
    C --> db

Loading

In this setup, client's connections would be proxied to the node that has the room loaded. It also frees all nodes from having to check if temporary rooms exist, because that's now deferred to the load balancers.

---
title: Joining a room - Happy Path
---
sequenceDiagram
    actor Alice
    participant OTT as OTT website
    participant L as Load Balancer
    participant N as Node
    Alice->>+OTT: Hello, I want to join "foo".
    OTT->>+L: Take this connection
    Note over L: Determines which node has "foo"
    L->>+N: Alice joins "foo"
    N->>-L: Joined
    L->>-OTT: Joined
    OTT->>-Alice: Joined
    Note over Alice,N: Alice's connection to OTT is transparently proxied to Node

Loading

---
title: Joining an unloaded room - Happy Path
---
sequenceDiagram
    actor Alice
    participant OTT as OTT website
    participant L as Load Balancer
    participant N as Node
    participant DB as Postgres
    Alice->>+OTT: Hello, I want to join "foo".
    OTT->>+L: Take this connection
    Note over L: Determines that no node has "foo"
    Note over L: Pick any node
    L->>+N: Does "foo" exist in the database? If so, load it.
    N->>+DB: find "foo" in Rooms table
    DB->>-N: found
    N->>-L: Yes, "foo" is now loaded.
    L->>+N: Alice joins "foo"
    N->>-L: Joined
    L->>-OTT: Joined
    OTT->>-Alice: Joined
    Note over Alice,N: Alice's connection to OTT is transparently proxied to Node

Loading

---
title: Generating and joining temporary room - Happy path
---
sequenceDiagram
    actor Alice
    participant OTT as OTT website
    participant L as Load Balancer
    participant N as Node
    Alice->>+OTT: Hello, I want to generate a room.
    OTT->>+L: Take this connection
    Note over L: Generating a room, no need to check database
    Note over L: Pick any node
    L->>+N: Generate a room
    Note over N: Initializes temporary room "foo"
    N->>-L: Loaded temporary room "foo"
    L->>Alice: Room is called "foo"
    Alice->>OTT: Join "foo"
    OTT->>L: Join "foo"
    L->>+N: Alice joins "foo"
    N->>-L: Joined
    L->>-OTT: Joined
    OTT->>-Alice: Joined
    Note over Alice,N: Alice's connection to OTT is transparently proxied to Node

Loading

---
title: Creating and Joining a new named temporary room - Edge case
---
sequenceDiagram
    actor Alice
    participant OTT as OTT website
    participant L as Load Balancer
    participant N as Node
    participant DB as Postgres
    Alice->>+OTT: Hello, I want to create "foo".
    OTT->>+L: Take this connection
    Note over L: Ensure that no node already has "foo"
    Note over L: Pick any node
    L->>+N: Does "foo" exist in the database?
    N->>+DB: find "foo" in Rooms table
    DB->>-N: not found
    Note over N: Initializes temporary room foo
    N->>-L: Loaded temporary room "foo"
    L->>+N: Alice joins "foo"
    N->>-L: Joined
    L->>-OTT: Joined
    OTT->>-Alice: Joined
    Note over Alice,N: Alice's connection to OTT is transparently proxied to Node

Loading

classDiagram
    class LoadBalancer {
        <<service>>
        nodes: Map
        clients: Vec~Client~
        
        find_loaded_room(string roomName) Option~&Node~
        least_loaded_node() &Node
        handle_websocket(WebSocket ws)
    }

    class Node {
        load: f64
        connection: Socket
        rooms: Set~string~
    }
    note for Node "represents an OttNode on the LoadBalancer"

    class OttNode {
        <<service>>
        rooms: Vec~Room~
        ...
    }
    note for OttNode "The actual OTT monolith"

    Node "1..*" -- "1..*" LoadBalancer
    Node "1" -- "1" OttNode

    class Room {
        users: Vec~RoomUser~
    }
    class RoomUser {
        client_id: String
        name: String
        balancer_id: String
    }
    class Client {
        client_id: String
        name: String
        room: String
        joined: bool
        socket: Socket
    }

    Room "1" -- "*" RoomUser
    LoadBalancer "1" -- "*" Client
    Client "1" .. "1" RoomUser
    OttNode "1" -- "*" Room

Loading

Joining a room

misc notes

• the browser websocket client sucks ass and you can barely do anything with it (say goodbye to any reasonable auth pipeline if you don't use cookies)

Scenarios that need to be handled

Migrate rooms off of a Monolith when it shuts down (graceful)

Assumes other Monoliths are available.

1. Monolith receives signal to shut down gracefully.
2. Monolith informs any single Balancer that it needs to move all rooms to other monoliths.
3. That Balancer distributes the rooms among the other monoliths according to the balancing strategy.
4. The Monoliths load room states from Redis.
5. The signaled Monolith disconnects from all balancers and shuts down.
6. Upon seeing the disconnected Monolith, all the Balancers search for new Monoliths that have the rooms loaded.
7. Balancers proceed with the Client Join sequence against the new Monoliths.

Changes for OTT

Info extractor

Not much needs to change for the info extractor pipeline, because it's stateless.

roommanager

The room manager would now be responsible for reporting what rooms are loaded.

Issues this would likely fix/contribute to

• Users not removed from user list when disconnected. #630