[Feature request] towards a better Python experimental example script

[nikohansen]

A most minimalistic example experiment looks like this

import cocoex  # experimentation module
import scipy.optimize  # to define the solver to be benchmarked

### input
suite_name = "bbob"
fmin = scipy.optimize.fmin  # optimizer to be benchmarked

### prepare
suite = cocoex.Suite(suite_name, "", "")
observer = cocoex.Observer(suite_name, "")

### go
for problem in suite:  # this loop may take several minutes or more
    problem.observe_with(observer)  # generates the data for cocopp
    fmin(problem, problem.initial_solution, disp=False)

Missing functionalities are (in order of supposed importance)

• restarts until enough successes are observed (like here)
• meaningful output folder name
• batch distribution (and accepting the batch number as argument)
• MKL bug fix?
• timing printout
• saving of solver output and termination flags for later investigation
• saving of setting when called with parameters (in particular budget_multiplier)

We may have several "examples" with increasing functionality (and complexity). How many seems a good number? Currently we have two, example_experiment_for_beginners and example_experiment2 (and also example_experiment_non_anytime for benchmarking non-anytime algorithms).

[nikohansen]

I have the following examples now, the next generally includes the functionality of the previous.
Starting from the script with 35 lines without the file docstring and

• restarts until enough successes are observed or the budget is exhausted (like here)
• meaningful output folder name

2. 65 lines (+30)
   • batch distribution and reading the batch number as script argument
   • saving of setting when called with parameters (in particular budget_multiplier)
3. 77 lines (+12)
   • timings printout
4. 84 lines (+7)
   • saving of solver output like the termination flags for later investigation
5. 122 lines (+38)
   • code for multiple solvers and space to implement calling code for another solver
6. 123 lines (+1) + file mkl_bugfix.py
   • MKL bug fix (the bug seems to have bugged users recently)

For reference, the example_experiment2.py had 160 lines.

[nikohansen]

The suggestion by @brockho and @ttusar is to have a separate file for the MKL bug fix and two scripts, one with 35 lines and one with full functionality.

Name suggestion: example_experiment_simple.py, example_experiment_complete.py.