GRPC Web Proxy to enable Browser Integration and Eliminate Electron IPC

[CMCDragonkai]

Specification

We can separate the networking to 3 layers:

• P2P application layer - e.g. kademlia, automerge, nodes domain
• RPC layer - e.g. grpc (and once considered jsonrpc but it's too late now)
• Data Transfer layer - UTP, Wireguard, QUIC

The RPC layer is what enables us to have rich communication protocols for:

• Unary
• Client Streaming, Server Streaming and Duplex Streaming

At the moment we have incorporated GRPC into js-polykey with the usage of these libraries:

    "@grpc/grpc-js": "^1.2.5",
    "google-protobuf": "^3.14.0",
    "@types/google-protobuf": "^3.7.4",
    "grpc_tools_node_protoc_ts": "^5.1.3",

This is working great for Agent to Agent and Client to Agent communication. However there are some challenges:

• Browsers do not currently have a way of calling the GRPC protocol at all - this means browser extensions cannot make use of the same API that we use in our client service
• PK GUI which is built on electron uses Electron IPC in the main Node process as a bridge between the renderer Vue application that runs in Chrome and the PK Agent. This adds an extra level of indirection (increasing latency) and work that should not have been needed.

There is a solution called GRPC-web. It's discussed here: https://grpc.io/blog/state-of-grpc-web/

The basic idea is to use a separate client library for the browser that uses browser-native protocols to perform GRPC operations.

There are some challenges here:

• The GRPC protocol is limited when used with grpc-web. The 2 implementations of it appears to lack both client streaming and duplex streaming. That means only unary and server streaming calls are available. This is not a huge issue since I don't think we have any calls on the client service that needs to use client streaming nor duplex streaming.
• The GRPC-Web does require a separate client library and thus separately generated stubs. The https://github.com/improbable-eng/grpc-web and https://github.com/grpc/grpc-web#client-configuration-options show how you need to generate additional marshaling code. Note that the official grpc/grpc-web implementation is the closest to how we already generate marshaling code in scripts/proto-generate.sh, it's just an additional output.
• GRPC-Web also requires a "proxy" that proxies GRPC-Web to normal GRPC. There are several in-process proxy implementations in Java and Golang. However none for Nodejs. It is ideal, somewhat necessary that we have an in-process proxy to avoid having to run separate process. We can take a look at existing implementations to see how we would implement our own in-process proxy.

Having GRPC-Web supported and an in-process proxy would mean:

• Being able to support browser extensions without the need of an additional API like Third Party Integration - HTTP API, OAuth2 Provider API, Plugin API, Library vs Framework, Smart Tokens #166.
• Eliminating the Electron IPC bridge entirely allowing the PK GUI Vue renderer application to directly talk to PK Agent

Additional context

• https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-WEB.md - this explains how the GRPC web protocol works
• Inter-op test for the Go middleware (from Improbable) grpc/grpc-web#91 and Publish interop information with grpc/grpc-web improbable-eng/grpc-web#162 - discussions on the interoperability between the 2 proxy implementations and 2 grpc-web clients
• https://github.com/grpc/grpc-web/blob/master/doc/in-process-proxy.md - How an in-process proxy should be structured
• https://github.com/improbable-eng/grpc-web/tree/master/go/grpcwebproxy - a stand alone grpc web proxy process that can be used for testing
• https://github.com/improbable-eng/grpc-web/tree/master/go/grpcweb - A go library that wraps the grpc go server as a grpc web server - this code is where we should draw inspiration from as a guide to implementing our own proxy
• Node-based proxy grpc/grpc-web#6 - A discussion and early attempt at writing an in-process Node proxy for grpc-web.
• Third Party Integration - HTTP API, OAuth2 Provider API, Plugin API, Library vs Framework, Smart Tokens #166 - having a GRPC-web proxy reduces the need to have an HTTP API

Note that there are multiple transports available in the libraries: https://github.com/improbable-eng/grpc-web/blob/master/client/grpc-web/docs/transport.md#built-in-transports. It seems to claim that websocket based transports actually enable both client and duplex streaming, so I'm guessing there's no need to ever go down the route of HTTP2 based transports and just focus on web socket transports?

At any case, for our usecase as we expect a limited deployment environment (browser extensions on latest browsers), it's fine to support only 1 transport.

Tasks

1. ...
2. ...
3. ...

[CMCDragonkai]

Once we can generate grpc-web stubs and have an in-process proxy. This means the grpc-web stubs should also be exported by js-polykey library to be usable.

[CMCDragonkai]

Current discussions about how to port over the grpcwebproxy to be Node-native: deeplay-io/nice-grpc#25 (comment)

Primary concern is to be able to run on the same port. We already use TLS.

[CMCDragonkai]

This is no longer relevant. With #249 we are now using JSON and our transport agnostic RPC. Which means browser's can just use websockets to contact our RPC system same as our client service.

[CMCDragonkai]

See #495 for converting to json RPC and @tegefaulkes we still need a new issue for your websocket work.