Transmitting a 4 GiB message crashes

[pgab]

In our application we experienced recently failures. Upon investigation this seems to be related to the message size which clearly exceeds 4 GiB. Yet the produced error messages are not very helpful. I've create an MVE at https://github.com/pgab/grpc-crash/tree/bytes.

I've ported the same example to hyperium/tonic that fails similarly but at least gives a proper error message. The MVE is at https://github.com/pgab/grpc-crash/tree/tonic.

Of course our application should take care of the large message and possibly use a stream in order to transport the message.

[stepancheg]

Neither gRPC nor protobuf designed to handle message of size 4G.

gRPC protocol message length is encoded in 4 bytes (thus cannot exceed 4G).

Although protobuf does not explicitly limit message size, the typical encoding of the message in protobuf is length-delimited, and length is signed int32 IIRC.

Quoting the document:

Protocol Buffers are not designed to handle large messages. As a general rule of thumb, if you are dealing in messages larger than a megabyte each, it may be time to consider an alternate strategy.

That said, both rust-protobuf and grpc-rust are misbehaving: they should clearly report the error instead of panicking.

Thank you for providing a good example.