[Feature]: Shared remote storage via Remote Storage API

[yurishkuro]

Requirement

I would like to use in-memory storage (or any other single-node storage) while still running multiple independent Jaeger components like collector/ingester/query.

Problem

Today in-memory storage can only be used in all-in-one, because it cannot be shared between different Jaeger components. This forces us to use expensive (slow to start) storage implementations in various CI pipelines.

Proposal

Since we now have Remote Storage gRPC API (a special mode of grpc-plugin storage type - #3383), a single process can be run as a remote storage backed by any other actual storage, including single-node backends like in-memory or Badger.

Proposal is to build a dedicated binary remote-storage that can be run separately, e.g.:

# run shared storage
$ go run ./cmd/remote-storage
{"level":"info","ts":...,"caller":"app/server.go:141","msg":"Starting GRPC server","addr":":17271"}

# point all-in-one to it as a simple test
$ SPAN_STORAGE_TYPE=grpc-plugin go run -tags ui ./cmd/all-in-one/main.go --grpc-storage.server=localhost:17271

Milestones

• new binary Add remote-storage service #3836
• [Feature]: publish binary and containers for remote-storage #3839
• Change crossdock end-to-end test to use "fast to start" storage #1221
• add documentation Add description of remote-storage component documentation#584