Third Party Integration - HTTP API, OAuth2 Provider API, Plugin API, Library vs Framework, Smart Tokens

[joshuakarp]

Created by @CMCDragonkai

Once we reintroduce OAuth2 into Polykey, because our OAuth2 requirements are pretty simple, it would be beneficial to implement it from scratch to reduce dependency requirements.

We have already implemented an Oauth2 client side from scratch in MR 141.

This would mean these dependencies we can get rid of:

    "oauth2orize": "^1.11.0",
    "passport-oauth2-client-password": "^0.1.2",
    "@types/oauth2orize": "^1.8.8",
    "@types/passport-oauth2-client-password": "^0.1.2",

Actually @CMCDragonkai is in favour of getting rid of password related libraries entirely so that we can be much lighterweight.

Like all of these:

    "passport": "^0.4.1",
    "passport-http": "^0.3.0",
    "passport-http-bearer": "^1.0.1",
    "@types/passport-http": "^0.3.8",
    "@types/passport-http-bearer": "^1.0.36",

In our usage of OAuth2 server-side, we would expect only a 2-legged flow. For example a "client-credentials-flow": https://docs.microsoft.com/en-us/linkedin/shared/authentication/client-credentials-flow

@CMCDragonkai : A client credentials flow is alot easier. And we should expect to just keep track of tokens, which we can use our vault system to store again.

[CMCDragonkai]

In addition to rebuilding OAuth2 server from scratch, there's a related problem about having an HTTP API.

GRPC is compatible with HTTP API if there's a gateway that translates our GRPC calls to HTTP.

Having an HTTP API will make it easier for Polykey agent to be integrated with other kinds of clients like Web browsers which don't support GRPC.

Right now the GRPC ecosystem has a thing called "grpc-gateway" which is a go-based server that generates HTTP API from a common protobuf spec with GRPC. However this is written in Go and cannot be integrated easily into our Node ecosystem.

We can manually create an HTTP API and call the same internal domain methods but this does add additional maintenance cost as we now have to maintain conformance between the GRPC API and the HTTP API.

Note we have past issue about using fastify instead of express for implementing an HTTP API: https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/issues/204

Therefore:

• Having an OAuth2 Server on Polykey is about allowing third party integration into PK Agent
• To have an OAuth2 ability, you must have an HTTP API for PK agent
• HTTP APIs will need to be manually implemented unless we ported grpc-gateway to JS/TS
• Should use minimal dependencies to do so like fastify rather than express
• HTTP API should be using OpenAPI for documentation
• HTTP API should be built using the official mapping of protobuf to json and vice versa https://developers.google.com/protocol-buffers/docs/proto3#json
• Developing an HTTP API solves the browser integration problem and third party integration problem that only has HTTP support
• Examples of other projects:
  • https://github.com/grpc-ecosystem/grpc-gateway
  • https://docs.microsoft.com/en-us/aspnet/core/grpc/httpapi?view=aspnetcore-5.0
  • https://github.com/google/gnostic
  • https://github.com/avast/grpc-json-bridge
• Note that this is different from grpc web gateway, which allows grpc running on browser to call GRPC but be proxied as HTTP requests, this about having a native HTTP API

[CMCDragonkai]

I'm going to elevate this issue into an Epic, as it is quite relevant multiple potential subissues.

• Native HTTP API for polykey-agent
• How to maintain conformance between GRPC and HTTP API
• Ensuring that this works for browsers to allow browser extensions in the future
• OAuth2 as an authorisation and authentication system to Polykey
• How this integrates with the existing SessionManagement we have

[CMCDragonkai]

I'm removing these dependencies and they deserve a review on whether they are needed to implement this:

    "express": "^4.17.1",
    "express-openapi-validator": "^4.0.4",
    "express-session": "^1.17.1",
    "js-yaml": "^3.3.0",
    "jsonwebtoken": "^8.5.1",
    "oauth2orize": "^1.11.0",
    "passport": "^0.4.1",
    "passport-http": "^0.3.0",
    "passport-http-bearer": "^1.0.1",
    "passport-oauth2-client-password": "^0.1.2",
    "swagger-ui-express": "^4.1.4",
    "@types/express-session": "^1.17.0",
    "@types/js-yaml": "^3.12.5",
    "@types/jsonwebtoken": "^8.5.0",
    "@types/oauth2orize": "^1.8.8",
    "@types/passport-http": "^0.3.8",
    "@types/passport-http-bearer": "^1.0.36",
    "@types/passport-oauth2-client-password": "^0.1.2",
    "@types/swagger-ui-express": "^4.1.2",
    "swagger-node-codegen": "^1.6.3",

There are way too many dependencies just to implement a HTTP API from the old code. Any new code should focus on something like this:

1. An http router server - something simpler than express, perhaps fastify or 0http - as minimal as possible because we are doing very basic things here in our http server, we just want to route it like the handlers we have for our grpc client server
2. The ability to implement an oauth2 server for authentication/authorisation, because we have JWT being used already via jose, this is something we should be able to implement ourselves directly without any additional dependencies (our jwt tokens should mean that our authentication is stateless, and it is expected to be put on the HTTP request headers)
3. A swagger api generator and swagger api outputter - this again should be as minimal as possible and perhaps fastify would be better https://github.com/fastify/fastify-swagger

[CMCDragonkai]

Here is the old code that can be revitalised when we investigate this:

HttpApi.ts

import fs from 'fs';
import net from 'net';
import path from 'path';
import jsyaml from 'js-yaml';
import { promisify } from 'util';
import Logger from '@matrixai/logger';

/** Internal */
import {
  PK_BOOTSTRAP_HOSTS,
  PK_NODE_PORT_HTTP,
  PK_NODE_ADDR_HTTP,
} from '../config';

import { Address } from '../nodes/Node';
import { TLSCredentials } from '../nodes/pki/PublicKeyInfrastructure';

// import passport from 'passport';
// import http from 'http';
// import https from 'https';
// import session from 'express-session';
// import swaggerUI from 'swagger-ui-express';
// import { Strategy as ClientPasswordStrategy } from 'passport-oauth2-client-password';
// import { BasicStrategy } from 'passport-http';
// import express, { RequestHandler } from 'express';
// import * as utils from './AuthorizationServer/utils';
// import * as config from './AuthorizationServer/Config';
// import * as OpenApiValidator from 'express-openapi-validator';
// import { User, Client } from './AuthorizationServer/OAuth2Store';
// import OAuth2 from './AuthorizationServer/OAuth2';
// import { Strategy as BearerStrategy } from 'passport-http-bearer';

class HttpApi {
  private openApiPath: string;
  private logger: Logger;

  private updateApiAddress: (apiAddress: Address) => void;
  private handleCSR: (csr: string) => string;
  private getRootCertificate: () => string;
  private getCertificateChain: () => string[];
  private getTlsCredentials: () => TLSCredentials;
  private getVaultNames: () => string[];
  private newVault: (vaultName: string) => Promise<void>;
  private deleteVault: (vaultName: string) => Promise<void>;
  private listSecrets: (vaultName: string) => string[];
  private getSecret: (vaultName: string, secretName: string) => Buffer;
  private newSecret: (
    vaultName: string,
    secretName: string,
    secretContent: Buffer,
  ) => Promise<void>;
  private deleteSecret: (
    vaultName: string,
    secretName: string,
  ) => Promise<void>;

  private tlsCredentials: TLSCredentials;
  private oauth: OAuth2;
  private expressServer: express.Express;
  private httpServer: http.Server;

  constructor(
    updateApiAddress: (apiAddress: Address) => void,
    handleCSR: (csr: string) => string,
    getRootCertificate: () => string,
    getCertificateChain: () => string[],
    getTlsCredentials: () => TLSCredentials,
    getVaultNames: () => string[],
    newVault: (vaultName: string) => Promise<void>,
    deleteVault: (vaultName: string) => Promise<void>,
    listSecrets: (vaultName: string) => string[],
    getSecret: (vaultName: string, secretName: string) => Buffer,
    newSecret: (
      vaultName: string,
      secretName: string,
      secretContent: string | Buffer,
    ) => Promise<void>,
    deleteSecret: (vaultName: string, secretName: string) => Promise<void>,
    logger?: Logger,
  ) {
    this.openApiPath = path.join(__dirname, '../openapi.yaml');
    this.updateApiAddress = updateApiAddress;
    this.handleCSR = handleCSR;
    this.getRootCertificate = getRootCertificate;
    this.getCertificateChain = getCertificateChain;
    this.getTlsCredentials = getTlsCredentials;
    this.getVaultNames = getVaultNames;
    this.newVault = newVault;
    this.deleteVault = deleteVault;
    this.listSecrets = listSecrets;
    this.getSecret = getSecret;
    this.newSecret = newSecret;
    this.deleteSecret = deleteSecret;
    this.logger = logger ?? new Logger();
  }

  async stop() {
    if (this.httpServer) {
      this.logger.info('Shutting down HTTP server');
      await promisify(this.httpServer.close)();
    }
  }

  async start(host = PK_NODE_ADDR_HTTP, port = parseInt(PK_NODE_PORT_HTTP)) {
    return new Promise<number>((resolve, reject) => {
      try {
        this.tlsCredentials = this.getTlsCredentials();
        this.oauth = new OAuth2(
          this.tlsCredentials.keypair.public,
          this.tlsCredentials.keypair.private,
          // this.logger.getLogger('OAuth2'),
          this.logger,
        );
        this.expressServer = express();

        this.expressServer.set('view engine', 'ejs');
        // Session Configuration
        const MemoryStore = session.MemoryStore;
        this.expressServer.use(
          session({
            saveUninitialized: true,
            resave: true,
            secret: 'secret',
            store: new MemoryStore(),
            cookie: { maxAge: 3600000 * 24 * 7 * 52 },
          }),
        );

        this.expressServer.use(express.json());
        this.expressServer.use(express.text());
        this.expressServer.use(express.urlencoded({ extended: false }));

        // create default client and user for the polykey node (highest priviledge)
        this.oauth.store.saveClient(
          'polykey',
          utils.createUuid(),
          ['admin'],
          true,
        );
        this.oauth.store.saveUser(
          'polykey',
          'polykey',
          utils.createUuid(),
          ['admin'],
          true,
        );

        this.expressServer.use(passport.initialize());
        this.expressServer.use(passport.session());

        // redirect from base url to docs
        this.expressServer.get('/', (req, res) => {
          res.redirect('/docs');
        });

        passport.use(
          'clientBasic',
          new BasicStrategy((clientId, clientSecret, done) => {
            try {
              const client = this.oauth.store.getClient(clientId);
              client.validate(clientSecret);
              done(null, client);
            } catch (error) {
              done(null, false);
            }
          }),
        );
        /**
         * BearerStrategy
         *
         * This strategy is used to authenticate either users or clients based on an access token
         * (aka a bearer token).  If a user, they must have previously authorized a client
         * application, which is issued an access token to make requests on behalf of
         * the authorizing user.
         *
         * To keep this example simple, restricted scopes are not implemented, and this is just for
         * illustrative purposes
         */
        passport.use(
          'accessToken',
          new BearerStrategy((token, done) => {
            try {
              const accessToken = this.oauth.store.getAccessToken(token);
              const user = this.oauth.store.getUser(accessToken.userId!);
              done(null, user, { scope: accessToken.scope ?? [] });
            } catch (error) {
              done(null, false);
            }
          }),
        );

        /**
         * Client Password strategy
         *
         * The OAuth 2.0 client password authentication strategy authenticates clients
         * using a client ID and client secret. The strategy requires a verify callback,
         * which accepts those credentials and calls done providing a client.
         */
        passport.use(
          'clientPassword',
          new ClientPasswordStrategy((clientId, clientSecret, done) => {
            try {
              const client = this.oauth.store.getClient(clientId);
              client.validate(clientSecret);
              done(null, client);
            } catch (error) {
              done(null, false);
            }
          }),
        );

        // Register serialialization and deserialization functions.
        //
        // When a client redirects a user to user authorization endpoint, an
        // authorization transaction is initiated.  To complete the transaction, the
        // user must authenticate and approve the authorization request.  Because this
        // may involve multiple HTTPS request/response exchanges, the transaction is
        // stored in the session.
        //
        // An application must supply serialization functions, which determine how the
        // client object is serialized into the session.  Typically this will be a
        // simple matter of serializing the client's ID, and deserializing by finding
        // the client by ID from the database.
        passport.serializeUser((user: User, done) => {
          done(null, user.id);
        });

        passport.deserializeUser((id: string, done) => {
          try {
            const user = this.oauth.store.getUser(id);
            done(null, user);
          } catch (error) {
            done(error);
          }
        });

        // token endpoints
        this.expressServer.post('/oauth/token', this.oauth.token);
        this.expressServer.post('/oauth/refresh', this.oauth.token);
        this.expressServer.get('/oauth/tokeninfo', [
          passport.authenticate(['accessToken'], { session: true }),
          this.oauth.tokenInfo.bind(this.oauth),
        ]);
        this.expressServer.get(
          '/oauth/revoke',
          this.oauth.revokeToken.bind(this.oauth),
        );

        // OpenAPI endpoints
        const schema = jsyaml.load(
          fs.readFileSync(this.openApiPath).toString(),
        );
        this.expressServer.get('/spec', (req, res) => {
          res.type('json').send(JSON.stringify(schema, null, 2));
        });
        this.expressServer.use(
          '/docs',
          swaggerUI.serve,
          swaggerUI.setup(schema, undefined, {
            oauth: {
              clientId: 'polykey',
            },
          }),
        );

        this.expressServer.use(
          OpenApiValidator.middleware({
            apiSpec: schema,
            validateResponses: true,
          }),
        );
        this.setupOpenApiRouter();

        // Start the server
        const pkHost = PK_BOOTSTRAP_HOSTS ?? 'localhost';
        const httpsOptions: https.ServerOptions = {
          cert: this.tlsCredentials.certificate,
          key: this.tlsCredentials.keypair.private,
          ca: this.tlsCredentials.rootCertificate,
        };

        this.httpServer = https
          .createServer(httpsOptions, this.expressServer)
          .listen({ port, host }, () => {
            const addressInfo = this.httpServer.address() as net.AddressInfo;
            const address = Address.fromAddressInfo(addressInfo);
            address.updateHost(pkHost);
            this.updateApiAddress(address);

            this.logger.info(
              `HTTP API endpoint: https://${address.toString()}`,
            );
            this.logger.info(
              `HTTP API docs: https://${address.toString()}/docs/`,
            );

            resolve(port);
          });
      } catch (error) {
        reject(error);
      }
    });
  }

  getOAuthClient(): Client {
    return this.oauth.store.getClient('polykey');
  }

  listOAuthTokens(): string[] {
    return Array.from(this.oauth.store.accessTokenStore.keys());
  }

  newOAuthToken(scopes: string[] = [], expiry = 3600): string {
    const expiryDate = new Date(Date.now() + expiry * 1000);
    const token = utils.createToken(
      this.oauth.store.privateKey,
      config.token.expiresIn,
      'polykey',
    );
    this.oauth.store.saveAccessToken(
      token,
      expiryDate,
      'polykey',
      'polykey',
      scopes,
    );
    return token;
  }

  revokeOAuthToken(token: string): boolean {
    this.oauth.store.deleteAccessToken(token);
    return !this.oauth.store.hasAccessToken(token);
  }

  // === openapi endpoints === //
  private handleRootCertificateRequest: RequestHandler = async (req, res) => {
    try {
      const response = this.getRootCertificate();
      this.writeString(res, response);
    } catch (error) {
      this.writeError(res, error);
    }
  };

  private handleCertificateChainRequest: RequestHandler = async (req, res) => {
    try {
      const response = this.getCertificateChain();
      this.writeStringList(res, response);
    } catch (error) {
      this.writeError(res, error);
    }
  };

  private handleCertificateSigningRequest: RequestHandler = async (
    req,
    res,
  ) => {
    try {
      const body = req.body;
      const response = this.handleCSR(body);
      this.writeString(res, response);
    } catch (error) {
      this.writeError(res, error);
    }
  };

  private handleVaultsListRequest: RequestHandler = async (req, res) => {
    try {
      const response = this.getVaultNames();
      this.writeStringList(res, response);
    } catch (error) {
      this.writeError(res, error);
    }
  };

  private handleNewVaultRequest: RequestHandler = async (req, res) => {
    try {
      const vaultName = (req as any).openapi.pathParams.vaultName;
      await this.newVault(vaultName);
      this.writeSuccess(res);
    } catch (error) {
      this.writeError(res, error);
    }
  };

  private handleDeleteVaultRequest: RequestHandler = async (req, res) => {
    try {
      const vaultName = (req as any).openapi.pathParams.vaultName;
      await this.deleteVault(vaultName);
      this.writeSuccess(res);
    } catch (error) {
      this.writeError(res, error);
    }
  };

  private handleSecretsListRequest: RequestHandler = async (req, res) => {
    try {
      const vaultName = (req as any).openapi.pathParams.vaultName;
      const response = this.listSecrets(vaultName);
      this.writeStringList(res, response);
    } catch (error) {
      this.writeError(res, error);
    }
  };

  private handleGetSecretRequest: RequestHandler = async (req, res) => {
    try {
      const vaultName = (req as any).openapi.pathParams.vaultName;
      const secretName = (req as any).openapi.pathParams.secretName;
      const response = this.getSecret(vaultName, secretName);

      const accepts = req.accepts()[0];
      if (!accepts || accepts == 'text/plain' || accepts == '*/*') {
        this.writeString(res, response.toString());
      } else if (accepts == 'application/octet-stream') {
        this.writeBinary(res, secretName, response);
      } else {
        throw Error(`MIME type not supported: ${accepts}`);
      }
    } catch (error) {
      this.writeError(res, error);
    }
  };

  private handleNewSecretRequest: RequestHandler = async (req, res) => {
    try {
      const vaultName = (req as any).openapi.pathParams.vaultName;
      const secretName = (req as any).openapi.pathParams.secretName;

      let secretContent: Buffer;
      const contentType = req.headers['content-type'];
      if (contentType == 'text/plain') {
        secretContent = Buffer.from(req.body);
      } else if (contentType == 'application/octet-stream') {
        secretContent = await new Promise<Buffer>((resolve, reject) => {
          const bufferList: Buffer[] = [];
          req.on('data', (data) => bufferList.push(data));
          req.on('error', (err) => reject(err));
          req.on('end', () => resolve(Buffer.concat(bufferList)));
        });
      } else {
        throw Error(`MIME type not supported: ${contentType}`);
      }

      await this.newSecret(vaultName, secretName, secretContent);
      this.writeSuccess(res);
    } catch (error) {
      this.writeError(res, error);
    }
  };

  private handleDeleteSecretRequest: RequestHandler = async (req, res) => {
    try {
      const vaultName = (req as any).openapi.pathParams.vaultName;
      const secretName = (req as any).openapi.pathParams.secretName;
      await this.deleteSecret(vaultName, secretName);
      this.writeSuccess(res);
    } catch (error) {
      this.writeError(res, error);
    }
  };

  // === Helper methods === //
  private writeSuccess(res: http.ServerResponse) {
    res.writeHead(200);
    res.end();
  }

  private writeError(res: http.ServerResponse, error: Error) {
    res.writeHead(500, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({ error: error.message }, null, 2));
  }

  private writeString(res: http.ServerResponse, text: string) {
    res.writeHead(200, { 'Content-Type': 'text/plain' });
    res.end(text);
  }

  private writeStringList(res: http.ServerResponse, list: string[]) {
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify(list, null, 2));
  }

  private writeJson(res: http.ServerResponse, payload: Record<string, any>) {
    res.writeHead(200, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify(payload, null, 2));
  }

  private writeBinary(
    res: http.ServerResponse,
    filename: string,
    payload: Buffer,
  ) {
    res.writeHead(200, {
      'Content-Type': 'application/octet-stream',
      'Content-Disposition': `file; filename="${filename}"`,
    });
    res.end(payload, 'binary');
  }

  private checkScope(scope: string[]) {
    return (req, res, next) => {
      // access control middleware to check for required scope
      if (!scope.some((r) => req.authInfo.scope.includes(r))) {
        res.statusCode = 403;
        return res.end('Forbidden');
      }
      return next();
    };
  }

  /**
   * The purpose of this route is to collect the request variables as defined in the
   * OpenAPI document and pass them to the handling controller as another Express
   * middleware. All parameters are collected in the requet.swagger.values key-value object
   *
   * The assumption is that security handlers have already verified and allowed access
   * to this path. If the business-logic of a particular path is dependant on authentication
   * parameters (e.g. scope checking) - it is recommended to define the authentication header
   * as one of the parameters expected in the OpenAPI/Swagger document.
   *
   * Requests made to paths that are not in the OpernAPI scope
   * are passed on to the next middleware handler.
   */
  private setupOpenApiRouter() {
    // setup all endpoints
    ///////////////////////////
    // Certificate Authority //
    ///////////////////////////
    this.expressServer.get('/ca/root_certificate', [
      passport.authenticate(['accessToken'], { session: true }),
      this.handleRootCertificateRequest.bind(this),
    ]);
    this.expressServer.get('/ca/certificate_chain', [
      passport.authenticate(['accessToken'], { session: true }),
      this.handleCertificateChainRequest.bind(this),
    ]);
    this.expressServer.post('/ca/certificate_signing_request', [
      passport.authenticate(['accessToken'], { session: true }),
      this.checkScope(['admin', 'request_certificate']),
      this.handleCertificateSigningRequest.bind(this),
    ]);
    ////////////
    // Vaults //
    ////////////
    this.expressServer.get('/vaults', [
      passport.authenticate(['accessToken'], { session: true }),
      this.checkScope(['admin', 'write_vaults', 'read_vaults']),
      this.handleVaultsListRequest.bind(this),
    ]);
    this.expressServer.post('/vaults/:vaultName', [
      passport.authenticate(['accessToken'], { session: true }),
      this.checkScope(['admin', 'write_vaults']),
      this.handleNewVaultRequest.bind(this),
    ]);
    this.expressServer.delete('/vaults/:vaultName', [
      passport.authenticate(['accessToken'], { session: true }),
      this.checkScope(['admin', 'write_vaults']),
      this.handleDeleteVaultRequest.bind(this),
    ]);
    /////////////
    // Secrets //
    /////////////
    this.expressServer.get('/vaults/:vaultName', [
      passport.authenticate(['accessToken'], { session: true }),
      this.checkScope(['admin', 'write_secrets', 'read_secrets']),
      this.handleSecretsListRequest.bind(this),
    ]);
    this.expressServer.get('/secrets/:vaultName/:secretName', [
      passport.authenticate(['accessToken'], { session: true }),
      this.checkScope(['admin', 'write_secrets', 'read_secrets']),
      this.handleGetSecretRequest.bind(this),
    ]);
    this.expressServer.post('/secrets/:vaultName/:secretName', [
      passport.authenticate(['accessToken'], { session: true }),
      this.checkScope(['admin', 'write_secrets']),
      this.handleNewSecretRequest.bind(this),
    ]);
    this.expressServer.delete('/secrets/:vaultName/:secretName', [
      passport.authenticate(['accessToken'], { session: true }),
      this.checkScope(['admin', 'write_secrets']),
      this.handleDeleteSecretRequest.bind(this),
    ]);
  }
}

export default HttpApi;

Config.ts

const token = {
  expiresIn: 60 * 60,
  calculateExpirationDate: () => new Date(Date.now() + 60 * 60 * 1000),
};

const refreshToken = {
  expiresIn: 52560000,
};

export { token, refreshToken };

OAuth2Store.ts

import {
  ErrorAuthCodeUndefined,
  ErrorInvalidCredentials,
  ErrorInvalidSecret,
  ErrorTokenUndefined,
  ErrorUserUndefined,
  ErrorClientUndefined,
} from '../../errors';
import { createUuid } from './utils';

class AuthorizationCode {
  code: string;
  clientId: string;
  redirectURI: string;
  userId: string;
  scope: string[];

  constructor(
    code: string,
    clientId: string,
    redirectURI: string,
    userId: string,
    scope: string[],
  ) {
    this.code = code;
    this.clientId = clientId;
    this.redirectURI = redirectURI;
    this.userId = userId;
    this.scope = scope;
  }
}
class AccessToken {
  token: string;
  expiration: Date;
  userId?: string;
  clientId?: string;
  scope?: string[];
  constructor(
    token: string,
    expiration: Date,
    userId: string,
    clientId: string,
    scope: string[],
  ) {
    this.token = token;
    this.expiration = expiration;
    this.userId = userId;
    this.clientId = clientId;
    this.scope = scope;
  }
}

class Client {
  id: string;
  private secret: string;
  scope: string[];
  trusted: boolean;
  constructor(
    id: string,
    secret: string,
    scope: string[] = [],
    trusted = false,
  ) {
    this.id = id;
    this.secret = secret;
    this.scope = scope;
    this.trusted = trusted;
  }

  updateSecret(secret: string) {
    this.secret = secret;
  }

  public get Secret(): string {
    return this.secret;
  }

  validate(secret: string) {
    if (this.secret != secret) {
      throw new ErrorInvalidSecret('secret does not match');
    }
  }
}

class User {
  id: string;
  username: string;
  private password: string;
  scope: string[];
  trusted: boolean;
  constructor(
    id: string,
    username: string,
    password: string,
    scope: string[] = [],
    trusted = false,
  ) {
    this.id = id;
    this.username = username;
    this.password = password;
    this.scope = scope;
    this.trusted = trusted;
  }

  updatePassword(password: string) {
    this.password = password;
  }

  public get Password(): string {
    return this.password;
  }

  validate(password: string) {
    if (this.password != password) {
      throw new ErrorInvalidCredentials('password does not match');
    }
  }
}

class OAuth2Store {
  accessCodeStore: Map<string, AuthorizationCode>;
  accessTokenStore: Map<string, AccessToken>;
  refreshTokenStore: Map<string, AccessToken>;
  clientStore: Map<string, Client>;
  userStore: Map<string, User>;

  publicKey: string;
  privateKey: string;

  constructor(publicKey: string, privateKey: string) {
    this.accessCodeStore = new Map();
    this.accessTokenStore = new Map();
    this.refreshTokenStore = new Map();
    this.clientStore = new Map();
    this.userStore = new Map();

    this.publicKey = publicKey;
    this.privateKey = privateKey;
  }

  ////////////////////////
  // Authorization Code //
  ////////////////////////
  hasAuthorizationCode(code: string): boolean {
    return this.accessCodeStore.has(code);
  }

  getAuthorizationCode(code: string): AuthorizationCode {
    if (!this.accessCodeStore.has(code)) {
      throw new ErrorAuthCodeUndefined('authorization code does not exist');
    }
    return this.accessCodeStore.get(code)!;
  }

  saveAuthorizationCode(
    code: string,
    clientId: string,
    redirectURI: string,
    userId: string,
    scope: string[],
  ): void {
    this.accessCodeStore.set(
      code,
      new AuthorizationCode(code, clientId, redirectURI, userId, scope),
    );
  }

  deleteAuthorizationCode(code: string): AuthorizationCode {
    const ac = this.getAuthorizationCode(code);
    this.accessCodeStore.delete(code);
    return ac;
  }

  ///////////////////
  // Access Tokens //
  ///////////////////
  hasAccessToken(token: string): boolean {
    return this.accessTokenStore.has(token);
  }

  getAccessToken(token: string): AccessToken {
    if (!this.accessTokenStore.has(token)) {
      throw new ErrorTokenUndefined('access token does not exist');
    }
    return this.accessTokenStore.get(token)!;
  }

  saveAccessToken(
    token: string,
    expiration: Date,
    userId: string,
    clientId: string,
    scope: string[] = [],
  ): AccessToken {
    this.accessTokenStore.set(
      token,
      new AccessToken(token, expiration, userId, clientId, scope),
    );
    return this.accessTokenStore.get(token)!;
  }

  deleteAccessToken(token: string): AccessToken {
    const at = this.getAccessToken(token);
    this.accessTokenStore.delete(token);
    return at;
  }

  ////////////////////
  // Refresh Tokens //
  ////////////////////
  hasRefreshToken(token: string): boolean {
    return this.refreshTokenStore.has(token);
  }

  getRefreshToken(token: string): AccessToken {
    if (!this.refreshTokenStore.has(token)) {
      throw new ErrorTokenUndefined('refresh token does not exist');
    }
    return this.refreshTokenStore.get(token)!;
  }

  saveRefreshToken(
    token: string,
    expiration: Date,
    userId: string,
    clientId: string,
    scope: string[] = [],
  ): AccessToken {
    this.refreshTokenStore.set(
      token,
      new AccessToken(token, expiration, userId, clientId, scope),
    );
    return this.refreshTokenStore.get(token)!;
  }

  deleteRefreshToken(token: string): AccessToken {
    const rt = this.getRefreshToken(token);
    this.refreshTokenStore.delete(token);
    return rt;
  }

  /////////////
  // Clients //
  /////////////
  hasClient(id: string): boolean {
    return this.clientStore.has(id);
  }

  getClient(id: string): Client {
    if (!this.clientStore.has(id)) {
      throw new ErrorClientUndefined('client does not exist');
    }
    return this.clientStore.get(id)!;
  }

  saveClient(
    id: string = createUuid(),
    secret: string,
    scope?: string[],
    trusted?: boolean,
  ): void {
    this.clientStore.set(id, new Client(id, secret, scope, trusted));
  }

  updateClient(
    id: string,
    secret?: string,
    scope?: string[],
    trusted?: boolean,
  ): void {
    const client = this.getClient(id);
    if (secret) {
      client.updateSecret(secret);
    }
    if (scope) {
      client.scope = scope;
    }
    if (trusted) {
      client.trusted = trusted;
    }
    this.clientStore.set(client.id, client);
  }

  deleteClient(id: string): Client {
    const client = this.getClient(id);
    this.clientStore.delete(id);
    return client;
  }

  //////////
  // User //
  //////////
  hasUser(id: string): boolean {
    return this.userStore.has(id);
  }

  getUser(id: string): User {
    if (!this.userStore.has(id)) {
      throw new ErrorUserUndefined('user does not exist');
    }
    return this.userStore.get(id)!;
  }

  findUserByUsername(username: string): User {
    const values = Array.from(this.userStore.values());
    if (values.findIndex((v) => v.username == username) == -1) {
      throw new ErrorUserUndefined('user does not exist');
    }
    return values.find((v) => v.username == username)!;
  }

  saveUser(
    id: string = createUuid(),
    username: string,
    password: string,
    scope?: string[],
    trusted?: boolean,
  ): void {
    this.userStore.set(id, new User(id, username, password, scope, trusted));
  }

  updateUser(
    username: string,
    password?: string,
    scope?: string[],
    trusted?: boolean,
  ): void {
    const user = this.findUserByUsername(username);
    if (password) {
      user.updatePassword(password);
    }
    if (scope) {
      user.scope = scope;
    }
    if (trusted) {
      user.trusted = trusted;
    }
    this.userStore.set(user.id, user);
  }

  deleteUser(id: string): User {
    const user = this.getUser(id);
    this.userStore.delete(id);
    return user;
  }
}

export default OAuth2Store;
export { AuthorizationCode, AccessToken, Client, User };

OAuth2.ts

import passport from 'passport';
import Logger from '@matrixai/logger';
import * as utils from './utils';
import * as config from './Config';
import oauth2orize from 'oauth2orize';
import Validation from './Validation';
import OAuth2Store from './OAuth2Store';

class OAuth2 {
  store: OAuth2Store;
  private server: oauth2orize.OAuth2Server;
  private validation: Validation;
  private expiresIn = { expires_in: config.token.expiresIn };
  private logger: Logger;

  constructor(publicKey: string, privateKey: string, logger: Logger) {
    this.store = new OAuth2Store(publicKey, privateKey);
    this.server = oauth2orize.createServer();
    this.validation = new Validation(this.store);
    this.logger = logger;

    /**
     * Exchange client credentials for access tokens.
     */
    this.server.exchange(
      oauth2orize.exchange.clientCredentials(async (client, scope, done) => {
        try {
          const token = utils.createToken(
            this.store.privateKey,
            config.token.expiresIn,
            client.id,
          );
          const expiration = config.token.calculateExpirationDate();
          // Pass in a null for user id since there is no user when using this grant type
          const user = this.store.findUserByUsername(client.id);
          const accessToken = this.store.saveAccessToken(
            token,
            expiration,
            user.id,
            client.id,
            scope,
          );
          done(null, accessToken.token, undefined, this.expiresIn);
        } catch (error) {
          done(error, false);
        }
      }),
    );

    this.server.serializeClient((client, done) => {
      done(null, client.id);
    });

    this.server.deserializeClient((id, done) => {
      try {
        const client = this.store.getClient(id);
        done(null, client);
      } catch (error) {
        done(error, null);
      }
    });
  }

  tokenInfo(req, res) {
    try {
      const accessToken = this.validation.tokenForHttp(req.query.access_token);
      this.validation.tokenExistsForHttp(accessToken);
      const client = this.store.getClient(accessToken.clientId!);
      this.validation.clientExistsForHttp(client);
      const expirationLeft = Math.floor(
        (accessToken.expiration.getTime() - Date.now()) / 1000,
      );
      res.status(200).json({ audience: client.id, expires_in: expirationLeft });
    } catch (error) {
      this.logger.error(error.toString());
      res.status(500).json({ error: error.message });
    }
  }

  revokeToken(req, res) {
    try {
      let accessToken = this.validation.tokenForHttp(req.query.token);
      accessToken = this.store.deleteAccessToken(accessToken.token);
      if (!accessToken) {
        accessToken = this.store.deleteRefreshToken(req.query.token);
      }
      this.validation.tokenExistsForHttp(accessToken);
      res.status(200).json({});
    } catch (error) {
      res.status(500).json({ error: error.message });
    }
  }

  public get token() {
    return [
      passport.authenticate(['clientBasic', 'clientPassword'], {
        session: true,
      }),
      this.server.token(),
      this.server.errorHandler(),
    ];
  }
}

export default OAuth2;

utils.ts

import jwt from 'jsonwebtoken';
import { v4 as uuid } from 'uuid';

function createUuid(): string {
  return uuid();
}

function createToken(privateKey: string, expiry = 3600, subject = ''): string {
  const token = jwt.sign(
    {
      jti: createUuid(),
      subject,
      exp: Math.floor(Date.now() / 1000) + expiry,
    },
    privateKey,
    {
      algorithm: 'RS256',
    },
  );

  return token;
}

function verifyToken(token: string, publicKey: string) {
  return jwt.verify(token, publicKey);
}

export { createUuid, createToken, verifyToken };

Validation.ts

import * as utils from './utils';
import * as config from './Config';
import OAuth2Store, { AuthorizationCode, Client } from './OAuth2Store';
import {
  ErrorClientUndefined,
  ErrorInvalidCredentials,
  ErrorInvalidToken,
  ErrorUserUndefined,
} from '../../errors';

class Validation {
  store: OAuth2Store;
  constructor(store: OAuth2Store) {
    this.store = store;
  }

  user(user, password) {
    this.userExists(user);
    if (user.password !== password) {
      throw new ErrorInvalidCredentials('User password does not match');
    }
    return user;
  }

  userExists(user) {
    if (user == null) {
      throw new ErrorUserUndefined('User does not exist');
    }
    return user;
  }

  clientExists(client) {
    if (client == null) {
      throw new ErrorClientUndefined('Client does not exist');
    }
    return client;
  }

  refreshToken(token, refreshToken, client) {
    utils.verifyToken(refreshToken, this.store.publicKey);
    if (client.id !== token.clientID) {
      throw new ErrorInvalidCredentials(
        'RefreshToken clientID does not match client id given',
      );
    }
    return token;
  }

  authCode(
    code: string,
    authCode: AuthorizationCode,
    client: Client,
    redirectURI: string,
  ) {
    utils.verifyToken(code, this.store.publicKey);
    if (client.id !== authCode.clientId) {
      throw new ErrorInvalidCredentials(
        'AuthCode clientID does not match client id given',
      );
    }
    if (redirectURI !== authCode.redirectURI) {
      throw new ErrorInvalidCredentials(
        'AuthCode redirectURI does not match redirectURI given',
      );
    }
    return authCode;
  }

  isRefreshToken(authCode: AuthorizationCode) {
    return authCode != null && authCode.scope.indexOf('offline_access') === 0;
  }

  generateRefreshToken(authCode: AuthorizationCode) {
    const refreshToken = utils.createToken(
      this.store.privateKey,
      config.refreshToken.expiresIn,
      authCode.userId,
    );
    const expiration = config.token.calculateExpirationDate();
    return this.store.saveRefreshToken(
      refreshToken,
      expiration,
      authCode.clientId,
      authCode.userId,
      authCode.scope,
    ).token;
  }

  generateToken(authCode: AuthorizationCode) {
    const token = utils.createToken(
      this.store.privateKey,
      config.token.expiresIn,
      authCode.userId,
    );
    const expiration = config.token.calculateExpirationDate();
    return this.store.saveAccessToken(
      token,
      expiration,
      authCode.userId,
      authCode.clientId,
      authCode.scope,
    ).token;
  }

  generateTokens(authCode: AuthorizationCode) {
    if (this.isRefreshToken(authCode)) {
      return Promise.all([
        this.generateToken(authCode),
        this.generateRefreshToken(authCode),
      ]);
    }
    return Promise.all([this.generateToken(authCode)]);
  }

  tokenForHttp(token: string) {
    try {
      utils.verifyToken(token, this.store.publicKey);
    } catch (error) {
      throw new ErrorInvalidToken('invalid_token');
    }
    let accessToken;
    accessToken = this.store.getAccessToken(token);
    if (!accessToken) {
      accessToken = this.store.getRefreshToken(token);
    }
    if (!accessToken) {
      throw new ErrorInvalidToken('token not found');
    }
    return accessToken;
  }

  /**
   * Given a token this will return the token if it is not null. Otherwise this will throw a
   * HTTP error.
   * @param   {Object} token - The token to check
   * @throws  {Error}  If the client is null
   * @returns {Object} The client if it is a valid client
   */
  tokenExistsForHttp(token) {
    if (!token) {
      throw new ErrorInvalidToken('invalid_token');
    }
    return token;
  }

  /**
   * Given a client this will return the client if it is not null. Otherwise this will throw a
   * HTTP error.
   * @param   {Object} client - The client to check
   * @throws  {Error}  If the client is null
   * @returns {Object} The client if it is a valid client
   */
  clientExistsForHttp(client) {
    if (!client) {
      throw new ErrorInvalidToken('invalid_token');
    }
    return client;
  }
}

export default Validation;

openapi.yaml

openapi: 3.0.2
info:
  title: Polykey API
  description: Peer to peer distributed secret sharing. HTTP API.
  version: 0.1.9
tags:
  - name: "ca"
    description: "Certificate authority operations"
  - name: "vaults"
    description: "Vault operations"
  - name: "secrets"
    description: "Secret Operations"
paths:
  /ca/root_certificate:
    get:
      tags:
        - "ca"
      summary: Returns the root certificate
      description: Returns the root certificate for the polykey node
      operationId: rootCertificate
      security:
        - bearerAuth: []
        - OAuth2-Client: []
      responses:
        "200":
          description: Root certificate
          content:
            text/plain:
              schema:
                type: string
        "401":
          description: Not authenticated
        "500":
          description: Internal server error
  /ca/certificate_chain:
    get:
      tags:
        - "ca"
      summary: Returns the certificate chain for verifying the root certificate
      description: Returns the certificate chain for the polykey node
      operationId: certificateChain
      security:
        - bearerAuth: []
        - OAuth2-Client: []
      responses:
        "200":
          description: Certificate Chain
          content:
            text/plain:
              schema:
                type: array
                items:
                  type: string
        "401":
          description: Not authenticated
        "500":
          description: Internal server error
  /ca/certificate_signing_request:
    post:
      tags:
        - "ca"
      summary: Request a signed certificate
      description: Request a certificate from the polykey node CA
      operationId: certificateSigningRequest
      security:
        - bearerAuth: [admin, request_certificate]
        - OAuth2-Client: [admin, request_certificate]
      requestBody:
        content:
          text/plain:
            schema:
              type: string
      responses:
        "200":
          description: Signed certificate
          content:
            text/plain:
              schema:
                type: string
        "401":
          description: Not authenticated
        "403":
          description: Access token does not have the required scope
        "500":
          description: Internal server error
  /vaults:
    get:
      tags:
        - "vaults"
      summary: List all vaults
      description: Returns a list of all vaults in the node
      operationId: vaultsList
      security:
        - bearerAuth: [admin, write_vaults, read_vaults]
        - OAuth2-Client: [admin, write_vaults, read_vaults]
      responses:
        "200":
          description: Vault List
          content:
            text/plain:
              schema:
                type: array
                items:
                  type: string
        "401":
          description: Not authenticated
        "403":
          description: Access token does not have the required scope
        "500":
          description: Internal server error
  "/vaults/{vaultName}":
    parameters:
      - name: vaultName
        description: Name of vault
        in: path
        required: true
        schema:
          type: string
    get:
      tags:
        - "secrets"
      summary: List secrets
      description: List all secrets in the vault named `vaultName`
      operationId: secretsList
      security:
        - bearerAuth: [admin, write_secrets, read_secrets]
        - OAuth2-Client: [admin, write_secrets, read_secrets]
      responses:
        "200":
          description: Secret List
          content:
            text/plain:
              schema:
                type: array
                items:
                  type: string
        "401":
          description: Not authenticated
        "403":
          description: Access token does not have the required scope
        "500":
          description: Internal server error
    post:
      tags:
        - "vaults"
      summary: Create a new vault
      description: Create a new vault named `vaultName`
      operationId: vaultsNew
      security:
        - bearerAuth: [admin, write_vaults]
        - OAuth2-Client: [admin, write_vaults]
      responses:
        "200":
          description: Vault was created successfully
        "401":
          description: Not authenticated
        "403":
          description: Access token does not have the required scope
        "500":
          description: Internal server error
    delete:
      tags:
        - "vaults"
      summary: Delete an existing vault
      description: Delete an existing vault called `vaultName`
      operationId: vaultsDelete
      security:
        - bearerAuth: [admin, write_vaults]
        - OAuth2-Client: [admin, write_vaults]
      responses:
        "200":
          description: Vault was deleted successfully
        "401":
          description: Not authenticated
        "403":
          description: Access token does not have the required scope
        "500":
          description: Internal server error
  "/secrets/{vaultName}/{secretName}":
    parameters:
      - name: vaultName
        description: Name of vault that contains the secret to be retrieved
        in: path
        required: true
        schema:
          type: string
      - name: secretName
        description: Name of secret to be retrieved
        in: path
        required: true
        schema:
          type: string
    get:
      tags:
        - "secrets"
      summary: Retrieve a secret
      description: Returns the secret `secretName` located in vault `vaultName`
      operationId: secretsGet
      security:
        - bearerAuth: [admin, write_secrets, read_secrets]
        - OAuth2-Client: [admin, write_secrets, read_secrets]
      responses:
        "200":
          description: Secret Content
          content:
            text/plain:
              schema:
                type: string
            application/octet-stream:
              schema:
                type: string
                format: binary
        "401":
          description: Not authenticated
        "403":
          description: Access token does not have the required scope
        "500":
          description: Internal server error
    post:
      tags:
        - "secrets"
      summary: Create a new secret
      description: Create a new secret within a specific vault
      operationId: secretsNew
      security:
        - bearerAuth: [admin, write_secrets]
        - OAuth2-Client: [admin, write_secrets]
      requestBody:
        description: Secret content
        content:
          text/plain:
            schema:
              type: string
          application/octet-stream:
            schema:
              type: string
              format: binary
      responses:
        "200":
          description: Secret was created successfully
          content: {}
        "401":
          description: Not authenticated
        "403":
          description: Access token does not have the required scope
        "500":
          description: Internal server error
    delete:
      tags:
        - "secrets"
      summary: Delete an existing secret
      description: Delete a new secret within a specific vault
      operationId: secretsDelete
      security:
        - bearerAuth: [admin, write_secrets]
        - OAuth2-Client: [admin, write_secrets]
      responses:
        "200":
          description: Secret was deleted successfully
        "401":
          description: Not authenticated
        "403":
          description: Access token does not have the required scope
        "500":
          description: Internal server error
components:
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
    OAuth2-Client:
      type: oauth2
      flows:
        clientCredentials:
          tokenUrl: /oauth/token
          refreshUrl: /oauth/refresh
          scopes:
            admin: Grants read and write access to both vaults and secrets
            request_certificate: Grants access to request a CA certificate
            write_vaults: "Grants delete, create and read access to vaults"
            read_vaults: Grants read access to vaults
            write_secrets: "Grants delete, create and read access to secrets"
            read_secrets: Grants read access to secrets

[CMCDragonkai]

Some extra thoughts on this after reading: https://news.ycombinator.com/item?id=28295348 and https://fly.io/blog/api-tokens-a-tedious-survey/.

• The tokens that we use to authenticate our CLI and GUI session should be the same as the tokens we are attempting for our HTTP API. That is the session manager and sessions domain is the starting point of this epic. Any development should evolve from that. Session Management Commands #204 Automatic session refresh and expiry information & Sessions Domain Refactoring According to Review #211
• There may be an issue with using JWT due to its kid parameter. Investigate this. We are currently using JWT for the session tokens, the notification messages and sigchain claims. The latter 2 aren't used for authentication, so it shouldn't apply here. But the need for JWT to be encoded and thus not human-readable is always an issue.
• Tokens are client-stored. They are not kept track of on the server side. Revocation is only possible globally by resetting the session key (this acts sort of like a version, if we assume a single PK agent is a single user). A similar issue can occur for these HTTP tokens. Alternatively a whitelist of tokens can kept stored as a traditional session DB. Or a blacklist of tokens can be stored similar to a certificate revocation list.
• Opening up PK to HTTP clients and third party users makes PK a multi-user system. Thus PK as a representative of a gestalt identity in a federated trust network goes 2nd order, while the CLI and GUI were designed from a single-user pov, this leads invariably to a multi-user pov, at least in terms of using the PK agent and its resources on behalf of the owner. In OAuth2 terms, there's only 1 resource owner and 1 resource-server/authorisation-server in PK, and many possible clients. Multi-user systems do require more fine-grained revocation abilities.
• With respect to OAuth2, other than our integration into identity providers, nobody signs into PK relying on a third party resource server. However there is a potential to sign into other client applications relying on PK as the resource server. We imagine that usually API tokens should be created directly in the API, CLI, GUI. But it is also possible that client applications want access to PK resources, and prompts their users to sign in via PK. The sign-in process for PK is just the standard session unlocking protocol, we can take inspiration from webauthn https://webauthn.me/. This implies however an HTTP redirection endpoint and a corresponding web-interface and public accessibility of the PK agent (and a fixed domain). I reckon this will probably be a very rare usecase, but it's worth considering how hashicorp vault expects client applications to acquire vault tokens in order to interact with vault.
• Vault/File schema - the existence of different kinds of API tokens means we have even more possible schema structures to support sophisticated secrets management for development workflow. Vault and File Schema - Ingress and Egress Schemas #222.
• Gestalt sync - creating an HTTP API token for one keynode means that other keynodes in the same gestalt can recognise that the token is valid within the gestalt. What does this mean? Can this be used to access resources across all keynodes in the same gestalt?
• Single sign-on (understood to be a gestalt), federated identity - Extending on the idea of signing-in to a client application via PK, it would mean a sort of federated identity system, since PK keynodes are part of a greater gestalt.