Multiple runs on single problem

[GoogleCodeExporter]

How to implement more than one run per problem=(function, instance, target)?

Original issue reported on code.google.com by [email protected] on 18 Jun 2014 at 8:35

[ttusar]

Is this still relevant? The experiments can be easily ran multiple times on the same problem instance, but what does this mean for post-processing?

[nikohansen]

As we have now a loop over problems in the example experiment (compared to a call like benchmark(optimizer, testbed, observer, budget)), this is solved in principle.

I believe it is a quite prominent use-case to pick a certain problem to use. Why? Because that's kind-of the point how to improve optimizers. Find a function where it doesn't work, understand why, improve the algorithm. The understand why part is where we need to run, possibly hundreds of, experiments on a single function.

In summary, it should be possible to pick a single problem and the howto should be easily accessible to a normal user.

In IPython it would work like:

In [34]: import cocoex as ex
In [35]: suite = ex.Suite("bbob", "", "")
In [36]: suite.  # TAB magic
suite.current_index         suite.find_problem_ids      suite.ids                   suite.name                  suite.options
suite.current_problem       suite.free                  suite.indices               suite.next_problem          suite.problem_names
suite.dimensions            suite.get_problem           suite.instance              suite.number_of_objectives  suite.reset
In [37]: suite.ids?
Type:        property
String form: <property object at 0x109f41cb0>
Docstring:
list of all problem IDs.

An ID can be used when calling the method `get_problem(id)`.

See also `find_problem_ids`.

In [38]: suite.find_problem_ids('f20', '10')  # looking for f20 in 10-D
Out[38]: []
In [39]: suite.find_problem_ids('20', '10')  # still looking for f20 in 10-D
Out[39]:
['bbob_f020_i01_d10',
 'bbob_f020_i02_d10',
 'bbob_f020_i03_d10',
 'bbob_f020_i04_d10',
 'bbob_f020_i05_d10',
 'bbob_f020_i51_d10',
 'bbob_f020_i52_d10',
 'bbob_f020_i53_d10',
 'bbob_f020_i54_d10',
 'bbob_f020_i55_d10',
 'bbob_f020_i56_d10',
 'bbob_f020_i57_d10',
 'bbob_f020_i58_d10',
 'bbob_f020_i59_d10',
 'bbob_f020_i60_d10',
 'bbob_f010_i01_d20',
 'bbob_f010_i02_d20',
 'bbob_f010_i03_d20',
 'bbob_f010_i04_d20',
 'bbob_f010_i05_d20',
 'bbob_f010_i51_d20',
 'bbob_f010_i52_d20',
 'bbob_f010_i53_d20',
 'bbob_f010_i54_d20',
 'bbob_f010_i55_d20',
 'bbob_f010_i56_d20',
 'bbob_f010_i57_d20',
 'bbob_f010_i58_d20',
 'bbob_f010_i59_d20',
 'bbob_f010_i60_d20']
In [40]: f = suite.get_problem('bbob_f020_i01_d10')
In [41]:

This use-case tells me that I would prefer the id bbob_f20_i01_d010 over bbob_f020_i01_d10 (minor thing), and that the attribute ids could be merged with the method find_problem_ids, because (suite.ids == suite.find_problem_ids()) is True, i.e. they return the same thing.

EDIT:

>>> suite = ex.Suite("bbob", "", "dimensions:10 instance_idx:1")
>>> f = suite.get_problem(19)
>>> f.id
'bbob_f020_i01_d10'

works as well :-) Not so bad. It might not work on new suites. As usual, I don't like the index shift.

[nikohansen]

The remaining question: what does/should the post-processing do with runs on a single function and/or even on the same instance?

[ttusar]

The suite filtering is independent of the suite, so it will work on new suites as well.

I'm not sure I understand the "remaining question". Should it do something different than it already does?

[nikohansen]

From above: "The current postprocessing code, though, does not take more than one run per
problem=(function, instance, target) into account if they are provided in different folders."

It's not exactly clear if this also applies to different subfolders. It probably works if they are in the same folder, but I wouldn't be sure this changed when additional consistency checks were introduced.

I didn't test the result of the post-processing in this case recently, so I can't say, for the time being, whether it should be different. I wouldn't be sure this works reasonably well, before I would have checked.

[brockho]

Highly related to #1117

[nikohansen]

Closed as covered by #1117.