Large RAM use for large messages

[pgab]

Our application uses some bigger messages. Those messages can get close to several gigabytes. We observed a fairly high RAM use related to those messages. I've created a test case at https://github.com/pgab/grpc-crash/tree/ram_use.

It clearly demonstrates that the more than double of RAM is being used on both server and client:

/usr/bin/time --verbose ../target/release/server --config config.toml 
^CCommand terminated by signal 2
	Command being timed: "../target/release/server --config config.toml"
	User time (seconds): 8.37
	System time (seconds): 3.55
	Percent of CPU this job got: 65%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:18.31
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 6293996
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 2359595
	Voluntary context switches: 115070
	Involuntary context switches: 978
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

and on the client side:

/usr/bin/time --verbose ../target/release/client-test
3072 MiB
	Command being timed: "../target/release/client-test"
	User time (seconds): 2.81
	System time (seconds): 4.02
	Percent of CPU this job got: 43%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.81
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 7344300
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 1
	Minor (reclaiming a frame) page faults: 1835317
	Voluntary context switches: 173080
	Involuntary context switches: 259
	Swaps: 0
	File system inputs: 48
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

[stepancheg]

Thank you for the report, sometimes grpc does unnecessary message copies/allocations. That need to be dealt with.