Use hyperfine and jq to improve evaluate.sh

[hundredwatt]

From #105

For lack of a better name, I called this script evaluate2.sh

$ ./evaluate2.sh
Usage: evaluate2.sh <fork name> (<fork name 2> ...)

[hundredwatt]

Note: I configured 1 warmup run... this differs from evaluate.sh so we could always change this to 0

[hundredwatt]

I added output of the raw times in addition to the trimmed means, eg:

fork,trimmed_mean
spullara,0.52814592528
royvanrijn,0.49758428661333337

fork,raw_times
spullara,0.51465599528,0.53491328628,0.52979903628,0.51972545328,0.53921453728
royvanrijn,0.49466057828000004,0.49655945328,0.49698236928,0.4992110372800001,0.50892974428

[gunnarmorling]

Very cool! One problem I see is that this moves the location of time measurement from just the java call to the launch script, which penalizes any contenders which call sdk for setting up a specific distro (as an example, I see the same time for merykitty as before, but +300ms for royvanrijn). We'd somehow have to extract this step.

[hundredwatt]

New look, based on: #105 (comment)

[hundredwatt]

@gunnarmorling I updated the script, please see my review comments below with some callouts and questions

[gunnarmorling]

@hundredwatt, thanks, I think this is getting into great shape. A few comments inline. Apart from those, is there a way we can capture the output of the contenders? We'd need that to compare that to the expected output (see process_output.java, which then of course wouldn't have to handle the output of time any more).

[hundredwatt]

@gunnarmorling Yes, we can do hyperfine --output <FILE>, I'll take a look at process_output.java shortly

[hundredwatt]

Addressed all comments from previous reviews

[gunnarmorling]

I'll take a look at process_output.java shortly

Excellent, thx! It's invoked via ./process.sh (and sorry for the code, I know it's a hot mess... ;)

[hundredwatt]

Remaining TODO:

• Save output to file
• Incorporate functionality from process_output.java

[AlexanderYastrebov]

I'll take a look at process_output.java shortly

BTW bash uses builtin time, the GNU /usr/bin/time supports TIME environment variable, see https://man7.org/linux/man-pages/man1/time.1.html

bash:

$ time sleep 1

real    0m1,009s
user    0m0,003s
sys     0m0,006s

/usr/bin/time has quite verbose default format which could be reduced just to real to avoid any kind of parsing.

$ /usr/bin/time sleep 1
0.00user 0.00system 0:01.00elapsed 0%CPU (0avgtext+0avgdata 2240maxresident)k
0inputs+0outputs (0major+78minor)pagefaults 0swaps

$ export TIME="%e"
$ /usr/bin/time sleep 1
1.00

Note also that default sh in Ubuntu is dash

~$ ll /bin/sh
lrwxrwxrwx 1 root root 4 Sep 27  2018 /bin/sh -> dash*

that does not have time builtin so all scripts that use #!/bin/sh and time (not /usr/bin/time) in Ubuntu print

0.00user 0.00system 0:01.00elapsed 0%CPU (0avgtext+0avgdata 2240maxresident)k
0inputs+0outputs (0major+78minor)pagefaults 0swaps

source sdk also does not work with dash.

I.e. for the next challenge baseline should stick to #!/bin/bash and use /usr/bin/time - then output could be configured via TIME :)

[hundredwatt]

process_output.java is used as follows:

1. Expected output is created via ./eval.sh baseline; cp baseline.out out_expected.txt
2. Fork output is created via ./eval.sh $fork
3. Invoke process.sh ./process.sh $fork
4. process_output.java then:
   a. Verifies the printed aggregation matches baseline
   b. Collects the times
   c. Computes the trimmed mean
   d. Prints the leaderboard line in Markdown table format

evaluate2.sh already handles b. and c., so we just need to add a. and d.

Let me know if I missed anything 😄

[gunnarmorling]

Yepp, that sounds exactly right 👍 .

[hundredwatt]

Ugh, hyperfine --output <FILE> overwrites FILE on each run... it doesn't append.

For now we'll only check the output of 1 run unless we can find a workaround

[gunnarmorling]

For now we'll only check the output of 1 run unless we can find a workaround

So if we run this script multiple times for multiple contenders, will we check the output of the last run for all contenders (ok)? Or just the last run of the last contender (bad)?

[gunnarmorling]

I've created #266 as a follow-up to this one, re-organizing the existing launch scripts to adhere to the structure established here.

[hundredwatt]

So if we run this script multiple times for multiple contenders, will we check the output of the last run for all contenders (ok)? Or just the last run of the last contender (bad)?

The former, we're ok 👍

[hundredwatt]

I pushed all the changes from process_output.java!

New look:

Happy Path

Verification failed

Going to do another round of testing on a Fedora box too

[hundredwatt]

For fun, the Leaderboard text now scans ./prepare_$fork.sh and extracts the Java version from sdk use if present 😄

[hundredwatt]

Oops, forgot the SMT / turbo stuff

[hundredwatt]

@gunnarmorling I think it's ready 😅

New look:

Happy Path

Verification failed

[hundredwatt]

What the warnings look like:

[gunnarmorling]

That's awesome, really great stuff, @hundredwatt! I'm gonna squash everything into one commit and merge it. We can do any necessary fine-tuning in follow-up PRs. Thanks a lot for pulling through with this one!

[gunnarmorling]

For fun, the Leaderboard text now scans ./prepare_$fork.sh and extracts the Java version from sdk use if present 😄

Wanted to suggest exactly that, but you beat me to it. Seems we can (re-)generate the leaderboard fully automatically with this change. Thanks again!

[hundredwatt]

My pleasure, happy to help out! Thanks for putting together such a fun, challenging and educational contest @gunnarmorling! Hopefully I’ll find some time to submit my own entry once we run out of infra things to improve 😂

…

On Tue, Jan 9, 2024 at 12:52 PM Gunnar Morling ***@***.***> wrote: Merged #182 <#182> into main. — Reply to this email directly, view it on GitHub <#182 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAWLOOOOTQJ3JJFL4JMSGLYNWNWVAVCNFSM6AAAAABBPYITNOVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJRGQZTIOBTGYYTONQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[AlexanderYastrebov]

Why do we need additional_build_steps_*.sh? I think the stuff could be in prepare_*.sh

They run one after another and there is a single use only:

• https://github.com/gunnarmorling/1brc/blob/main/additional_build_steps_thomaswue.sh
• https://github.com/gunnarmorling/1brc/blob/main/prepare_thomaswue.sh

I don't see why we can not mv additional_build_steps_thomaswue.sh prepare_thomaswue.sh and drop support for additional_build_steps_*.sh.