If you are tired of writing scripts manually, which instruct an application you want to benchmark, this program is what you are searching for. You give lists of command line parameter values to the Measurement Instructor and it executes your application with every possible combination of the given parameter set.
If you specify a name prefix for the output files on the command line, the standard output of your application executions will be saved appropriately. Generally you want to save the output files in an empty directory, as there can be a lot of them.
E.g. minstructor -o ./results "./binary --scheme foo --seed=range(3) --param [a,b]"
will result in executing the following commands:
./binary --scheme foo --seed=0 --param a > ./results/out_0.txt
./binary --scheme foo --seed=0 --param b > ./results/out_1.txt
./binary --scheme foo --seed=1 --param a > ./results/out_2.txt
./binary --scheme foo --seed=1 --param b > ./results/out_3.txt
./binary --scheme foo --seed=2 --param a > ./results/out_4.txt
./binary --scheme foo --seed=2 --param b > ./results/out_5.txt
You can specify ranges with various patterns:
Example | Type |
---|---|
[4,a,8,...] |
simple lists |
range(0,20,3) |
python-like ranges (start, end, step) |
linspace(0,2,5) |
python numpy-like linear ranges (start, stop, num) |
logspace(3,12,10,2) |
python numpy-like log ranges (start, stop, num, base) |
Probably you want to collect certain metrics of your application
executions and evaluate them. You can use the mcollector
to achieve
that efficiently. The mcollector
expects multiple files each
containing the stdout
of one application run. Your application should
output every relevant information. E.g. if you execute ./binary --scheme foo --seed=16547
, a stdout
processable by the mcollector
could look like:
...
- scheme -> foo
- bandwidth = 20 GB/s
foo bar baz ... weather: "sunny and warm"
footime: 0.4687 s
Here you can also write about the bandwidth or scheme etc.
unless you don't assign it twice.
- random-seed --> 16547
...
You can collect your results, which are saved in output files, in a CSV table with:
mcollector ./results/out_*
The mcollector
is able to recognize certain assignment patterns, like
they are shown above, and will extract the words or numerical values
after the keywords. It is important that the keywords are only
assigned once in each output file. E.g. if the shell expansion in the
example above results in several output files, an example CSV output of
the mcollector
could look like:
scheme,bandwidth,weather,footime,random-seed,data-file-path
foo,20,"sunny and warm",0.4687,16547,/abs/path/results/out_0.txt
foo,10,"rainy",N/A,1756,/abs/path/results/out_1.txt
foo,0,"windy and rainy",0.4864,1654,/abs/path/results/out_2.txt
If a file does not contain a keyword assignment, which is found in other files, the value is substituted with N/A.
To build the manual pages you need to have pandoc
, which can be
installed with most system package manager programs. I wrote the scripts
in Ruby, thus you need a Ruby implementation and the Ruby package
manager gem
to install the required RubyGems:
$ gem install progressbar
$ gem install test-unit # to run the tests
$ gem install rake # to run the tests and build the manual pages
If your shell does not find the RubyGems, it might be helpful to add
$(ruby -e 'print Gem.user_dir')/bin
to your PATH
environment
variable.
Up to know the installation (rake install
) will simply copy the
scripts to /usr/local/bin
. The manual pages are installed to $(man -w | cut -d: -f1)/man1
.
You will find more information on the manual pages, which can be built
with rake man
, or built and installed with rake install
.
Why I prefer minstructor
in comparison to the Google Benchmark library
https://github.com/google/benchmark
google-benchmark | minstructor |
---|---|
-less predefined range functions | +predefined numpy like range functions |
-long running jobs (error-prone) | +independent jobs |
\ | -many (temporary) output files |
-functions should have already been tested | +benchmarking and testing |
+good for real micro benchmarks | -not fast for benchmarks with timings similar to the prog. launch overhead |
-library dependency | -minstructor must also be installed |
-syntax understanding needs time | +mainly self explanatory |
-no slurm support | +multiple backends (also slurm) |
-functions must not have cout |
+functions should have various informative output |
-ouput to CSV: every benchmark must contain every self defined counter | |
-time measurement points must be set manually anyways as in most cases we do not want to measure the time of a whole function | |
-strong coupling between your application and the API of the library |