gRPC Collector Configurable Max Message Size

[js8080]

Well, I got past my other issue where I was trying to send messages over the NGINX Ingress default max message size (see related question #3196) but unfortunately, I seem to have hit another max message size limit that seems to be in the Collector itself.

On the Jaeger Client side, I set my max span size to over 4mb (a value > 410241024 which is the default max span size and also the default max message size for gRPC apparently). This allows me to attempt to send a larger Span to the gRPC Collector but when I turn on gRPC debugging, I see the following error:

I0810 20:33:37.251032 0 T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\ext\transport\chttp2\transport\parsing.cc:541: HTTP:7:TRL:CLI: grpc-message: 67 72 70 63 3a 20 72 65 63 65 69 76 65 64 20 6d 65 73 73 61 67 65 20 6c 61 72 67 65 72 20 74 68 61 6e 20 6d 61 78 20 28 34 31 39 38 34 39 38 20 76 73 2e 20 34 31 39 34 33 30 34 29 'grpc: received message larger than max (4198498 vs. 4194304)'

This was confounding me for a while but then I realized I was seeing this error in my Ingress Controller on my K3S cluster (NGINX):

2021-08-10T20:56:02.316034992-05:00 2021/08/11 01:56:02 [error] 538#538: *11132265 upstream rejected request with error 0 while reading response header from upstream, client: 10.100.175.180, server: jaeger-collector.my-container-dev.myorg.com, request: "POST /jaeger.api_v2.CollectorService/PostSpans HTTP/2.0", upstream: "grpc://10.42.0.25:14250", host: "jaeger-collector.my-container-dev.myorg.com:443"

From digging around online, it seems like his might be due to the upstream service. However, I don't see anything in my Collector pod logs even though I have the log-level set to debug:

  containers:
  - args:
    - ... (omitted others) ...
    - --log-level=debug

I've been digging through the gRPC documentation and the Jaeger Collector CLI options but I have not been able to find any way to set the max gRPC message size options (one of these two it seems but likely the second one):

• grpc.max_send_message_length
• grpc.max_receive_message_length

Is there some way that the Jaeger Collector allows me to set these max message sizes?

I don't know Go but from what I can tell in the Go gRPC library, one can set the Max Received Message Size (grpc.max_receive_message_length) by doing something like this (maybe in the grpc.go -> StartGRPCServer() function:

// StartGRPCServer based on the given parameters
func StartGRPCServer(params *GRPCServerParams) (*grpc.Server, error) {
	var server *grpc.Server

	var grpcOpts []grpc.ServerOption
	grpcOpts = append(grpcOpts, grpc.MaxRecvMsgSize(10*1024*1024))

That StartGRPCServer function gets called by collector.go -> Start() so that's my best guess and I also don't see MaxCallRecvMsgSize referenced anywhere else in the Jaeger code so I'm guessing it is just not a supported setting right now. I'd love it if someone could confirm!

I created an Issue (Bug) for this: gRPC Collector Does Not Support Configurable Max Message Size #3210