Benchmarking compiler's peak memory use

[cameel]

This PR adds an extra column showing maximum resident set size of the process during its lifetime, as reported by /usr/bin/time. In local benchmarks this is solc alone, while in external ones this is the whole forge command.

It also updates the scripts to report the CPU time rather than wall clock time. Since solc is single-threaded they should mostly match but CPU time is what we're really interested in (we could very easily start reporting both if that ever changes).

Also small tweaks, like time rounding in external benchmarks or cosmetic changes to how table looks.

[clonker]

Locally I get No such file or directory errors:

λ ~/dev/solidity/ memory-usage-in-benchmarks* test/benchmarks/local.sh 
|         File         | Pipeline | Bytecode size |   Time   | Memory (peak) | Exit code |
|----------------------|----------|--------------:|---------:|--------------:|----------:|
jq: error: Could not open file /tmp/solc-benchmark-VCQ2QW/time-and-status-legacy.txt: No such file or directory
jq: error: Could not open file /tmp/solc-benchmark-VCQ2QW/time-and-status-legacy.txt: No such file or directory
jq: error: Could not open file /tmp/solc-benchmark-VCQ2QW/time-and-status-legacy.txt: No such file or directory
| `verifier.sol`       |   legacy |       0 bytes |   0.00 s |         0 MiB |         0 |

the directory does contain files though

λ ~/dev/solidity/test/ memory-usage-in-benchmarks* ls -lah /tmp/solc-benchmark-VCQ2QW
total 4.0K
drwx------  2 mho  mho   100 Sep 24 17:40 .
drwxrwxrwt 30 root root 1020 Sep 24 17:46 ..
-rw-r--r--  1 mho  mho   144 Sep 24 17:40 benchmark-warn-err.txt
-rw-r--r--  1 mho  mho     0 Sep 24 17:40 bytecode-ir.bin
-rw-r--r--  1 mho  mho     0 Sep 24 17:40 bytecode-legacy.bin

Is there something wrong with my setup, do i have to source something beforehand?

[cameel]

Is there something wrong with my setup, do i have to source something beforehand?

No, it should be enough to just run the script the way you're doing it. external.sh does require running external-setup.sh beforehand, but local.sh is self-contained.

I think you just don't have GNU time installed. I'm getting the same message when I try using an executable that does not exist. Not sure about other distros, but at least on Arch Linux it comes from the time package, and is not included in the base group as one might expect.

I added a check against that so now you should at least be getting a more understandable error message.

[cameel]

I ran the benchmark on 0.8.24 and 0.8.27 release binaries to test it and to see how the memory usage changed. Looks like it decreased slightly in most cases.

I also included timing numbers to get some perspective on how far we got since then.

File	Pipeline	Time (0.8.24)	Time (0.8.27)	Memory (0.8.24)	Memory (0.8.27)	Exit code
openzeppelin	legacy	13 s	10 s	563 MiB	519 MiB	0
openzeppelin	ir	58 s	40 s	1071 MiB	1220 MiB	1
uniswap-v4	legacy	30 s	24 s	1114 MiB	1013 MiB	0
uniswap-v4	ir	467 s	157 s	5244 MiB	4805 MiB	0
eigenlayer	legacy	107 s	86 s	3280 MiB	2833 MiB	0
eigenlayer	ir	2304 s	716 s	22272 MiB	20346 MiB	0
seaport	legacy	215 s	183 s	5232 MiB	4476 MiB	0
seaport	ir	760 s	13 s	8020 MiB	1042 MiB	1
sablier-v2	legacy	174 s	155 s	5603 MiB	4882 MiB	0
sablier-v2	ir		935 s		25535 MiB	1

Note that the ones, which ended due to an error differ more, but I think they must have just crashed at different points. At least in case of OZ, the 0.8.27 run did finish while 0.8.24 didn't. For this reason I also didn't wait for sablier to finish on 0.8.24. It would probably run out of memory on my system anyway.

[clonker]

I added a check against that so now you should at least be getting a more understandable error message.

That was precisely the issue! It's still a relatively fresh installation on this machine, the package was indeed missing :)