Transport Agnostic RPC

[CMCDragonkai]

We have reviewed the gRPC API, and worked with it extensively. It's time to work out a better RPC layer for PK. There are several problems to to consider here:

• Cohesion
  • https://developers.google.com/protocol-buffers/docs/proto3
• Pagination
  • Pagination Deployment to Service Handlers and CLI Commands and Domain Collections #237
  • https://gitlab.com/MatrixAI/Engineering/Polykey/polykey-design/-/issues/22
• Streaming and unification with pagination
  • Client streaming can support some client usecases where only the client process has the data
• Authentication & Sessions - usage of metadata and bearer & basic tokens
  • Automatically Retry the CLI Command upon Session Unlocking #240
• Client and Server Interception (Middleware)
  • How client interception works: Session Bin Test Fixes #296 (comment)
  • Add server interceptors grpc/grpc-node#419
  • Client interceptors are available and are used in src/client
  • Server interceptors are not yet available, so instead we use DIed functions like authenticator
• CQRS in order to maintain consistency between different clients
  • https://gitlab.com/MatrixAI/Engineering/Polykey/polykey-design/-/issues/23
• Naming Standardisation
  • Standardize the naming of GRPC Functions/Commands and Message Types #218
• HTTP API portability and Web Gateway Portability
  • Third Party Integration - HTTP API, OAuth2 Provider API, Plugin API, Library vs Framework, Smart Tokens #166
  • GRPC Web Proxy to enable Browser Integration and Eliminate Electron IPC #235
  • Porting grpcwebproxy to native Node.js deeplay-io/nice-grpc#25
• Error handling
  • https://www.grpc.io/docs/guides/error/
  • We serialize our PK exceptions and then pass them to the client where it deserializes exceptions
  • Do we have to consider backwards compatibility here? What happens if the agent side serializes an exception that doesn't exist on the client side code, the client side should gracefully handle this by throwing a generic exception that indicates the client doesn't understand the exception (but provide exception details)
  • Resolve trailing metadata usage and sensitive information being exposed in exceptions Transport Agnostic RPC #249 (comment)
• Typescript Compatibility
  • Wrapper methods in GRPCClientClient to get parameter types of the wrapped GRPC call #200
• API versioning
  • How should PK API be versioned? Should we always try to maintain 100% backwards compatibility, or have some way of versioning incompatible changes?
  • How is a version signaled in gRPC? In REST this can be done in the URL or in the header metadata
  • Guide from Microsoft https://docs.microsoft.com/en-us/aspnet/core/grpc/versioning?view=aspnetcore-5.0
  • GRPC service version would be independent of the js-polykey version which is independent from the state version, which would give us 3 versions to manage:
    • pk source version - updated when source code changes, minor is new features, major is any new incompatibility from anywhere: cli, service, node state... etc, e.g. 1.0.0 -> 1.1.0 -> 2.0.0 -> 2.0.1
    • grpc service version - only major version and only changes on breaking changes, e.g. 1 -> 2 -> 3
    • node state version - only major version and only changes on breaking changes to the node state, e.g. 1 -> 2 -> 3

We have 2 main proto files:

1. proto/schemas/Client.proto
2. proto/schemas/Agent.proto

And a Test.proto as well, this will need to be used to generate the marshaling code.

Additional context

• React Native and Mobile OS (iOS and Android) Compatibility #155 - compatibility with Mobile operating systems
• Third Party Integration - HTTP API, OAuth2 Provider API, Plugin API, Library vs Framework, Smart Tokens #166 - having a transport agnostic RPC will make it easier for third party integration as it's possible to extend the different transport options to communicate with the PK agent - such as TCP, and UDP... etc
• GRPC Web Proxy to enable Browser Integration and Eliminate Electron IPC #235 - would not be a problem anymore if a web-based transport is enabled, either that or if electron directly bridges into the nodejs runtime (rather than having the FE call into a BE proxy) - this is also relevant to any browser-extensions
• Async API for Notifications & Delay-Tolerant Synchronisation #248 - some RPC functionality is intended to support asynchronous API, should consider these design requirements for notifications
• Using QUIC/HTTP3 to replace utp-native for the Data Transfer Layer in the networking domain #234 - our RPC should be compatible on the transport layer for P2P communication, P2P hole punching occurs underneath the RPC layer, so the P2P side has to bootstrap from a lower level communication protocol which is purely message oriented
• IPv6 Support #400 - the RPC should be agnostic to IPv4 or IPv6
• Asynchronous Promise Cancellation with Cancellable Promises, AbortController and Generic Timer #297 - RPC mechanism must be cancellable ideally using cancellable promises, which includes cancelling the underlying side effect
• Apply timedCancellable to Sigchain such as the deadline in nodes claim process #243 - it's a good idea for the RPC to have timeouts on its calls and to handle such timeouts, in particular we would want to be able to create "custom" protocols on top of any streams, or better would be to "lift" such protocols into the the underlying RPC system such as the nodes claiming process
• https://exploringjs.com/nodejs-shell-scripting/ch_web-streams.html - Explanation of how to use Web Streams

Tasks

1. - Review the proto3 guide, naming standard with reference to the proto files, look for ways to abstract or clean up the data
   • Creating .proto definition subdomains. #279
   • - Version the Client.proto and Agent.proto with version names and test out multiple-version services, and find out what the client does when the version wanted is not available
   • - Test out grpc reflection for versioning if necessary
   • - Update all the primitive types, and any usage of 64 bit int should have the string hint
   • - Add some benchmarks to the grpc, and check if MAX_CONCURRENT_CONNECTIONS is used
   • - Clean up types Wrapper methods in GRPCClientClient to get parameter types of the wrapped GRPC call #200 or gRPC abstractions based on new grpc libraries
   • - Consider how to shutdown the grpc server and terminate all client connections as well
2. - Figure out CQRS, pagination and streaming
   • - Clean up js-pagination client and server side utilities
   • - Pagination on unary calls, and streaming calls that can be initialised with a pagination parameters (which would enable some sort of CQRS and event sourcing concepts)
3. - Review in reference to authentication and sessions
   • - Fix Exceptions with Sensitive Data and Trailing Metadata Usage Transport Agnostic RPC #249 (comment)
4. - Check the HTTP API and web gateway compatibility
5. - Deal with the fact that the new grpc-js 1.4.1 version has some incompatibility: TypeError: http2Server.on is not a function.

[CMCDragonkai]

Note that all fields in proto3 is optional and this is all facilitated by a default value. https://stackoverflow.com/questions/31801257/why-required-and-optional-is-removed-in-protocol-buffers-3

The default values are all based on the types of the fields. So for primitives like numbers, it's 0, but for other subtypes, it is null in JS.

Optional fields returned in proto 3.5. What this means is usually generates a handler to check whether it was set at all, that way you can differentiate between a value not being set by the client versus the client setting the value to the default value. This can be important when you want to change behaviour if the client really didn't set the value.

with optional you get the ability to explicitly check if a field is set. Lets say you have a field int32 confidence. Currently when receiving a message which such a type you cannot know the difference between confidence = 0 or confidence not set. Because default values are optimized away in the serialization. If you mark the field as optional then presumably some extra bits are set in the serialization and a has_confidence() method will be generated so that you on the receiving end can disambiguate the two.

[CMCDragonkai]

Regarding all the number types in proto3. There are 64 bit numbers:

int32
int64
uint32
uint64
sint32
sint64
double
float

Due to this issue protocolbuffers/protobuf#3666. There's no support for bigint yet. The default loses precision.

Work around is:

message dummy { 
  uint64 bigInt = 1; [jstype = JS_STRING] 
}

This turns it into a string.

[CMCDragonkai]

This shows the mapping from the proto3 code to the generated JS code: https://developers.google.com/protocol-buffers/docs/reference/javascript-generated, of interest is the map, bytes, one of, and enums.

[CMCDragonkai]

The guide says that if we want to stop using a certain field in a message. It's important to reserve the field number and also the field name so that it cannot be used again. This means that older clients won't get confused.

Basically the field numbers don't have to be actually in sequence. They represent unique positions in the message.

message Foo {
  reserved 2, 15, 9 to 11;
  reserved "foo", "bar";
}

enum Foo {
  reserved 2, 15, 9 to 11, 40 to max;
  reserved "FOO", "BAR";
}

[CMCDragonkai]

Regarding pagination I like this usage of repeated:

message SearchResponse {
  repeated Result results = 1;
}

message Result {
  string url = 1;
  string title = 2;
  repeated string snippets = 3;
}

It seems that this makes it easier to define a single message type and then lists of messages. I had done this previously in OpenAPI when I had situations where you have GET /resource/1 vs GET /resources where the former acquires a single resource, and the latter fetches a page of resources.

I imagine that responses may have further metadata beyond just the core data. So I could imagine types being represented like:

// the actual domain structure
XDomainStructure {
  // ...
}

// a single resource /resources/1
YResponse {
  XDomainStucture x_domain = 1;
}

// multiple resources /resources
ZResponse {
  repeated XDomainStructure x_domains = 1;
}

That way "response" messages are differentiated from domain structures that we want to work with.

So in our proto definitions we can identify the relevant domain structures we want to work with, these should really be derived manually by the programmer by investigating each domain. For example in our notifications domain, we have a domain structure representing a notification message. But this is not the same type as the type of request & response messages being sent back and forth on the gRPC API about notifications.

[CMCDragonkai]

It's possible to also import proto files. This can make it easier to manage our proto definitions if we separate our type definitions from our message definitions from our service definitions. It could allow us to form subdirectories in src/proto/schemas. We would have to check if our CLI command scripts/proto-generate.sh can actually do this though.

Importing is just a matter of:

import "myproject/other_protos.proto";

To re-export, just use import public "other.proto";.

I believe the name should start from where the --proto_path is set to. This should be src/proto/schemas.

[CMCDragonkai]

I forgot if we were ever able to use any of the google protos. Like import "google/protobuf/any.proto";. The any type allows us to nest any kind of message type. It will be serialized as bytes. We are likely not going to use this particular type google.protobuf.Any though.

However there are a number of other types that are quite useful like google/protobuf/timestamp.proto. That's all located here: https://developers.google.com/protocol-buffers/docs/reference/google.protobuf

[CMCDragonkai]

The oneof can have properties that use different field numbers. But no matter what if you set any one of those properties, it should clear the other properties.

The map only works for fixed types for keys and values. So for arbitrary values, we should use JSON encoding and return it as a string.

The map also doesn't work as nested. So a value of map can't be another map.

Although you should be able to put another map inside a message type, and that would be sufficient to nest maps.

[CMCDragonkai]

Need to point out that re-reading https://medium.com/expedia-group-tech/the-weird-world-of-grpc-tooling-for-node-js-part-3-d994de02bedc seems to mean that our current usage of static tooling means that importing packages may not work. Not sure... at this point.

[CMCDragonkai]

This issue seems to indicate all we need to do is copy the proto source code verbatim in our src/proto/schemas and then it should work. agreatfool/grpc_tools_node_protoc_ts#72

[CMCDragonkai]

Note that a gRPC channel tends to have a limit number of max concurrent connections (from that point onwards rpc requests are queued). I believe we have separate channels to each different node. We aren't likely to hit this max number in production right now, but the chattiness may increase when we integrate gestalt synchronisation #190. This issue has more details grpc/grpc#21386. The current work around is to create new channels and load balance between channels, however this is considered to be a temporary solution.

Use a pool of gRPC channels to distribute RPCs over multiple connections (channels must have different channel args to prevent re-use so define a use-specific channel arg such as channel number).

This will impact any benchmarking we do that involves launching multiple concurrent gRPC requests.

This limit is mentioned here: https://github.com/grpc/grpc-node/blob/master/PACKAGE-COMPARISON.md and also compares the @grpc/grpc-js vs grpc library (that is now deprecated).