Rework benchmarking system

[felixhekhorn]

No description provided.

[codecov]

Codecov Report

Merging #68 into develop will decrease coverage by 31.23%.
The diff coverage is 50.00%.

@@             Coverage Diff              @@
##           develop      #68       +/-   ##
============================================
- Coverage    92.20%   60.96%   -31.24%     
============================================
  Files           19       19               
  Lines          821      825        +4     
============================================
- Hits           757      503      -254     
- Misses          64      322      +258     

Flag	Coverage Δ
#benchmarks	?
#regression	?
#unittests	60.96% <50.00%> (+1.77%)	⬆️

Impacted Files	Coverage Δ
src/yadism/runner.py	31.14% <0.00%> (-67.19%)	⬇️
src/yadism/observable_name.py	51.61% <66.66%> (-7.71%)	⬇️
src/yadism/structure_functions/f2_light.py	27.77% <0.00%> (-72.23%)	⬇️
src/yadism/coupling_constants.py	36.36% <0.00%> (-63.64%)	⬇️
src/yadism/output.py	31.25% <0.00%> (-56.25%)	⬇️
src/yadism/structure_functions/fl_light.py	44.44% <0.00%> (-55.56%)	⬇️
src/yadism/structure_functions/esf.py	42.85% <0.00%> (-53.97%)	⬇️
src/yadism/structure_functions/fl_heavy.py	55.00% <0.00%> (-45.00%)	⬇️
.../yadism/structure_functions/splitting_functions.py	55.55% <0.00%> (-44.45%)	⬇️
... and 6 more

[alecandido]

Towards a stable Benchmarks Suite

We are currently refactoring the whole benchmarking system (not regression test) to make it stable and suitable for CI workflows

The following steps are in our plan:

Redefine the db structure:

• apfel will have its own database, with its theories, observables and cache
• qcdnum as well, both theories and observables won't be the same of apfel, because of the different implemented features in the two codes
• logs will have their own db, used mainly locally (even if it will be produced also by the workflows)
• sandbox input db will be added to support the new structure

observables

at the same time we are going to differentiate observables per external program, but also unifying them in few records (essentially one per perturbative order), to be able to speed up the calculation either in the external objects (that will precompute everything on a grid) and in the same yadism, that will take advantage of caching system and numba JIT compilation

Less dependencies into the workflows

• the cache will be uploaded in the same way of what it is happening for regression, since it is even more stable than regression itself
• apfel computation won't be performed anymore during workflows
• qcdnum comparison will be available for workflows
• conda won't be anymore a part of the game

Welcome sandbox

• a new way of running comparisons will be available for local use only
• it will have its dedicated script ecosystem, to be able to change comparison in a fast and flexible way
• it won't affect any test/benchmark score so there is no reason to worry about its stability

[alecandido]

• think about list of known differences for make benchmarks aware of particular kinematical corners