Ensure Session token remains safe with multiple processes.

[tegefaulkes]

An MR has been created for this issue here: https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/212

Specification

The Session Session.writeToken in src/sessions/Session.ts:116 makes use of proper-lockfile. However we need to create a test to make sure this functions properly when multiple processes are attempting to update the token file.

The proper-lockfile library is fairly old, and is more complex than what we require. As such, we should switch to using fd-lock instead.

fd-lock only provides two methods: lock and unlock. There is no explicit method to check whether a file is already locked, however, since a file can only be locked if it is not already locked then we can simply attempt to lock the file - if we fail, the file is already locked.

Additional context

imagine 2 calls

pk dosomething1 # uses T1 token (T1 token changes to T2)
pk dosomething2 # uses T2 token

that's in the case of a serial calls but in parallel calls

pk dosomething1 & # uses T1 token (changes to T2 when it finishes)
pk dosomething2 & # uses T1 token as well 
(tries to change to T3, but drops the change when it realises that T2 locking

so at the end of parallel calls, the token state us still T2, but that's fine as T2 is still valid, and T3 is simply dropped

Information on fd-lock can be found here.

Tasks

1. Explore the use of fd-lock as an alternative to proper-lockfile
2. Create test for multiple processes attempting to update session and token file (in tests/client and/or tests/bin).
3. Confirm that the locking functions as expected.

[tegefaulkes]

Refer to https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/210#note_664718851 for some context.

[CMCDragonkai]

The primary thing to understand here is that the multi-process situation here is about the PK client, no the PK agent.

We have a client-server architecture now between clients and agents.

Every single CLI call pk ... is a separate execution of the PK client process.

This means there are going to be multiple PK client processes running simultaneously.

The PK GUI is also a PK client, it's just a long-running one.

Therefore we need to ensure that the session token usage by the PK clients are safe across multiple simultaneous process executions and that means we need simulation tests inside tests/client or tests/bin to ensure that our session tokens are not clobbered up.

[CMCDragonkai]

A diagram should be made for this and ported to the sessions wiki that shows the component of multiple clients, a single agent, and the usage of a common session key stored at the HOME directory or client state directory that is being used to authenticate.

A second component diagram can be used to represent a multi-step process (similar to a sequence diagram) but with labelled ordinal arrows.

[CMCDragonkai]

This should illustrate the entire architecture.

In addition to this, one should use the same diagram, but illustrate a labelled ordinal actions.

I have to redo the diagram cause asciiflow broke.. so the above is a snapshot.

The labelled ordinal action diagram is actually a "top-down" sequence diagram. It's the application of the sequence of steps to a component diagram. An example is this: https://datatracker.ietf.org/doc/html/rfc6749#section-1.2. The advantage of using this over a sequence diagram are:

• More clearer mapping of the sequence to the components - can aid mental mapping
• Is clearer for multiple components, whereas normal sequence diagram would have to cross lines (avoid line crossing in diagrams)

Disadvantage is:

• Cannot be used for very complex sequences, there will be too many arrows

So the normal sequence diagram is better for fewer agents and complex sequences and sequence nesting. While top down sequence is better for multiple agents, and in order to map the a high level sequence to multiple components.

[CMCDragonkai]

Because each Session shares the ~/.polykey/client/token, they need to use a lock to coordinate access. This lock is primarily intended for when the token is updated.

The token is updated on every single call. The PK agent returns a new valid token to be used.

In our case, we expect that any call holds a lock to the token, and if another process discovers that the lock is held, it simply drops and cancels the session token update request. That way the token is only updated by the first process that is starting a call, one the process finishes their call, they should release the token file lock. All of this logic should be inside the Session class.

[CMCDragonkai]

This issue is independent of #240, however this should be done first as it in relation to correctness of our system.

And #240 is an enhancement of the UX of using the CLI/GUI.

[CMCDragonkai]

Diagram fixed up and with extra details.

                                       ┌────────────────┐
                                       │ SessionManager │  SessionManager generates and verifies session tokens
Client on different host/user          └────────┬───────┘
                                                │
    ┌───────────────┐                   ┌───────┴──────┐
    │ PolykeyClient ├───────────────────► PolykeyAgent │   Agent accepts authenticated requests
    └───────┬───────┘                   └───────▲──────┘
            │                                   │
       ┌────┴────┐            ┌─────────────────┼─────────────────┐
       │ Session │            │                 │                 │
       └─────────┘    ┌───────┴───────┐ ┌───────┴───────┐ ┌───────┴───────┐
                      │ PolykeyClient │ │ PolykeyClient │ │ PolykeyClient │  Multiple CLI & GUI clients on same host/user
                      └───────┬───────┘ └───────┬───────┘ └───────┬───────┘
                              │                 │                 │
                         ┌────┴────┐       ┌────┴────┐       ┌────┴────┐
                         │ Session │       │ Session │       │ Session │     Session manages token lifecycle
                         └────┬────┘       └────┬────┘       └────┬────┘
                              │                 │                 │
                              └─────────────────┼─────────────────┘
                                                │
                                   ┌────────────┴────────────┐
                                   │ ~/.polykey/client/token │  Concurrency managed via file lock
                                   └─────────────────────────┘

                         Shared session token for authenticated requests

[CMCDragonkai]

The https://github.com/mafintosh/fd-lock library is a much simpler locking library. Since the concurrency management only requires you to check if the lock is in fact locked and not have to wait/block for the lock to be unlocked, then it can be a much simpler library to use instead of proper-lockfile. BTW I don't think proper-lockfile uses fd-lock.

The proper-lockfile is much older, bigger attack surface. So it's better to use something simpler.

[emmacasolin]

@CMCDragonkai I've remade your diagram above with sequence numbers included. This is my understanding of what safety of the session token should look like across multiple processes. Please let me know if I'm off about anything.

┌─────────────────────────────────────────┐
│              SessionManager             │
└──────▲─┬────────────────────────────────┘
       │ │
       │ │
┌──────┴─┴────────────────────────────────┐
│                                         ├────────────┐
│               PolykeyAgent              │            │
│                                         ◄─────────┐  │
└────┬─┬─┬────────────────────┬───────────┘         │  │
     │ │ │                    │                     │  │
     │ │ │                    │                     │  │
     1 2 3                    4                     6  │ Client on different
     │ │ │                    │                     │  │      host/user
     │ │ │                    │                     │  │
 ┌───┴─┴─┴───────┐        ┌───┴───────────┐        ┌┴──┴───────────┐
 │ PolykeyClient │        │ PolykeyClient │        │ PolykeyClient │
 └───┬─┬─┬───┬───┘        └───┬───▲───┬───┘        └───┬───▲───┬───┘
     │ │ │   │                │   │   │                │   │   │
    ┌┴─┴─┴───┴┐              ┌┴───┴───┴┐              ┌┴───┴───┴┐
    │ Session │              │ Session │              │ Session │
    └┬───┬───┬┘              └┬───┬───┬┘              └┬───┬───┬┘
     │   │   │                │   │   │                │   │   │
     │   │   │                │   │   │                │   │   │
     │   │   11               │   5   9                │   │   │
     │   │   │                │   │   │                7   8   10
     ▼   │   │                │   │   ▼                │   │   │
  (empty)│   │                │   │(locked)            │   │   │
     ┌───▼───▼────────────────▼───┴───┐                │   │   │
     │                                ◄────────────────┘   │   │
     │                                │                    │   │
     │    ~/.polykey/client/token     ├────────────────────┘   │
     │                                │                        │
     │                                │(locked)◄───────────────┘
     └────────────────────────────────┘

1. PolykeyAgent starts a new PolykeyClient and associated Session to run a command. The session checks ~/.polykey/client/token and finds either no token or an invalid token.
2. Session requests a new token from the SessionManager.
3. SessionManager provides a new token, which is passed back through to the Session, which puts it in ~/.polykey/client/token, locking the file. Now the PolykeyClient can begin running its asynchronous command.
4. While the first command is running a second command is initiated, so the PolykeyAgent initialises a second PolykeyClient and associated Session. This Session checks ~/.polykey/client/token and finds a valid token.
5. The valid token is passed to the second PolykeyClient and the second asynchronous command can also commence.
6. While both of the first two commands are still running, a Client on a different host sends an authenticated request to the PolykeyAgent.
7. The PolykeyAgent responds to this third request and the third PolykeyClient requests a token from its Session. The Session subsequently checks ~/.polykey/client/token for a valid token.
8. A valid token is returned and the third PolykeyClient can run its command.
9. The second PolykeyClient has now finished running its command and attempts to refresh the shared session token in ~/.polykey/client/token, however, since the file was locked by the first PolykeyClient's Session it drops this request to refresh the token.
10. Similarly, when the third PolykeyClient attempts to refresh the session token it also drops the request upon discovering that the file is locked.
11. The first PolykeyClient finishes running its command. It unlocks ~/.polykey/client/token and refreshes the token.

Note that the second and third PolykeyClients must also lock ~/.polykey/client/token while their own commands are running to prevent the token from being refreshed by any other command that finishes before their own. The session token should only be refreshed by the last Client to finish using it. In this sense, EVERY command needs to lock ~/.polykey/client/token when it starts and unlock it when it ends, so we need to be able to have multiple, independent locks on the session token such that it can only be refreshed when ALL of the locks are released.

[CMCDragonkai]

Just a quick comment, in this diagram, you don't need the other 2 PolykeyClient. Focus on just the interaction between client and agent. Then you can bring up the second client to show the locking interaction.

[emmacasolin]

So just like this? Or are the arrows labelled 4 and 5 also not necessary?

┌────────────────────────────────────────┐
│             SessionManager             │
└─────▲─┬────────────────────────────────┘
      │ │
      │ │
┌─────┴─┴────────────────────────────────┐
│                                        │
│              PolykeyAgent              │
│                                        │
└───┬─┬─┬────────────────────┬───────────┘
    │ │ │                    │
    │ │ │                    │
    1 2 3                    4
    │ │ │                    │
    │ │ │                    │
┌───┴─┴─┴───────┐        ┌───┴───────────┐
│ PolykeyClient │        │ PolykeyClient │
└───┬─┬─┬───┬───┘        └───┬───▲───┬───┘
    │ │ │   │                │   │   │
   ┌┴─┴─┴───┴┐              ┌┴───┴───┴┐
   │ Session │              │ Session │
   └┬───┬───┬┘              └┬───┬───┬┘
    │   │   │                │   │   │
    │   │   │                │   │   │
    │   │   7                │   5   6
    │   │   │                │   │   │
    ▼   │   │                │   │   ▼
 (empty)│   │                │   │(locked)
    ┌───▼───▼────────────────▼───┴───┐
    │                                │
    │                                │
    │    ~/.polykey/client/token     │
    │                                │
    │                                │
    └────────────────────────────────┘

[emmacasolin]

Also I had a look at fd-lock and I don't think it will work for our purposes. If a file is already locked then you can't lock it again, and there's no way to determine who locked the file and what processes are still using it. All it offers is locking and unlocking so we would need another way to track which processes are still reliant on the file before we unlock and refresh the token.

[CMCDragonkai]

As 2 separate diagrams.