Explore new serialization library used for communication between horaedb-server

[jiacai2050]

Describe This Problem

Our RemoteEngineService use gRPC to communicate between different servers, and adopt protobuf as encode/decode methods, protobuf is not a very high performance library, and we observe high CPU usage in our production env.

Proposal

Explore new serialization library, which support following features:

• Zero copy, which means its wire format is the same as memory layout.
• Native rust binding
• Support integration with tonic

Additional Context

Some choices:

• https://flatbuffers.dev/
• https://fury.apache.org/docs/start/usage#rust

[jiacai2050]

@zealchen

[zealchen]

@zealchen

Okay. Those libs look super awesome, let me build a benchmark to demonstrate performance first.

[jiacai2050]

Our benchmarks show flatbuffers has the best perf, and it can be integrated with tonic, so let's refactor our remote service.

https://github.com/apache/incubator-horaedb/blob/6baf6df0445f085dba4c330e12d019104483f2ee/src/server/src/grpc/remote_engine_service/mod.rs#L950

Currently we can only refactor the write API, since the CPU cost of it is the most obvious.