ResourceExhausted desc = grpc: received message larger than max

[lzh-lab]

Describe the bug
When alerts is increasing, I can't get all rules detail info
{ "status": "error", "data": null, "errorType": "server_error", "error": "unable to retrieve rules from other rulers, rpc error: code = ResourceExhausted desc = grpc: received message larger than max (54207607 vs. 4194304)" }

My cortex config

`auth_enabled: true

server:
http_listen_port: 9009

grpc_server_max_recv_msg_size: 104857600
grpc_server_max_send_msg_size: 104857600
grpc_server_max_concurrent_streams: 1000
`

[pstibrany]

Unfortunately you're hitting the limit on the client side. It defaults to 4 MB and currently Ruler doesn't expose options to change it.

Ruler exposes only TLS options:

cortex/pkg/ruler/ruler.go

Line 66 in e69d628

ClientTLSConfig tls.ClientConfig `yaml:"ruler_client"`

It should also expose other gRPC options:

cortex/pkg/util/grpcclient/grpcclient.go

Lines 104 to 107 in 7fe8ef6

	type ConfigWithTLS struct {
	GRPC Config `yaml:",inline"`
	TLS tls.ClientConfig `yaml:",inline"`
	}

Would you be interested in sending the PR? Changing one for another should be backward-compatible change.

[lzh-lab]

Unfortunately you're hitting the limit on the client side. It defaults to 4 MB and currently Ruler doesn't expose options to change it.

Ruler exposes only TLS options:

cortex/pkg/ruler/ruler.go

Line 66 in e69d628

ClientTLSConfig tls.ClientConfig `yaml:"ruler_client"`

It should also expose other gRPC options:

cortex/pkg/util/grpcclient/grpcclient.go

Lines 104 to 107 in 7fe8ef6

	type ConfigWithTLS struct {
	GRPC Config `yaml:",inline"`
	TLS tls.ClientConfig `yaml:",inline"`
	}

Would you be interested in sending the PR? Changing one for another should be backward-compatible change.

Is there any plan to support expose options to change it ？

[pstibrany]

I've sent PR #3523 to expose these options.

[pstibrany]

Note that PR also changes the limits, especially the one you're hitting is changed from 4 MiB to 100 MiB, so you may not even need to change those settings now.

[lzh-lab]

This is really good news.