Adding independent restarts as additional trials

[nikohansen]

We should consider to allow, or even encourage, to do additional trials of the very same instance instead of within-trial restarts. This would also have the advantage that restarts do not need to be logged additionally (see e.g. #1073).

First step: the post-processing must normalize the ECDF, the average RT, etc. by the number of same-instance repetitions. (EDIT: done and can be toggled via genericsettings.balance_instances) The normalization must be done such that first each (distinct) instance gets the same weight and then each problem gets the same weight in the result. In other words: we need to make sure that the post-processing can deal with data which have a different number of instances and/or same-instance repetitions appropriately.

Remark: this might also be a good step in the direction where instances are not anymore conceptually considered as the very same function / problem. This might indeed already be the better approach for (some functions in) the bbob-biobj test suite (see also #1018).

Open questions regarding the rule on the experimental setup: what are the eligible triggers for such additional trials, apart from the current budget? Could the trigger depend on the function or even the instance number without generating potential problems?

EDIT: for simulated restarts, there are two ways of instance data selection which lead potentially to different results:

• we select each instance with the same probability (which is currently not implemented) instead of each trial with the same probability. That is, trials from instances with many repetitions have a lower probability to be selected. This models the situation when instances reflect different restart positions all chosen with the same probability. Data won't bias this probability.
• we select a subset of instances such that for each (EDIT: unsuccessful!?) instance a similar (ideally the same) overall budget is used (the one corresponding to the smallest overall budget, but at least one trial). This "simulates" restarts as they are implemented in the example experiments. The selection is repeated for each run with simulated restarts.

Which one is better justifiable?

EDIT: changing the experimental recommendation for conditional instance repetitions can (and is somewhat likely to) "break" any other code that displays results, because an unweighted aggregation over all instances (e.g. an average) is now biased.

[DONE] EDIT: to avoid various issues, it seems advisable to repeat all instances of the same problem the same number of times. In this case, the used trigger seems rather uncritical.

[loshchil]

Not sure that "restarts" and their triggers should be in focus.
Instead, the platform should not be surprised to see several
data folders of the same instance, but just combine their data.
In this case, the user may perform 1 run or 100 runs if she/he
wishes to get "smoother" curves.

On Tue, Jul 26, 2016 at 3:48 PM, nikohansen [email protected]
wrote:

We should consider to allow, or even encourage, to do additional trials
of the very same instance instead of within-trial restarts. This would
also have the advantage that restarts do not need to be logged additionally
(see e.g. #1073 #1073).

First step: the post-processing must normalize the ECDF, the average RT,
etc. by the number of same-instance repetitions. In other words: we need to
make sure that the post-processing can deal with this kind of data
appropriately.

Remark: this might also be a good step in the direction where instances
are not anymore conceptually considered as the very same function /
problem.

Open questions: what are the eligible triggers for such restarts, apart
from the current budget? Could the trigger depend on the function or even
the instance number without generating potential problems?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#1117, or mute the thread
https://github.com/notifications/unsubscribe-auth/ADoe-6mcMDgVMy9hUIbAPFx58J458FUCks5qZhBHgaJpZM4JVKek
.

[nikohansen]

The instance-wise normalization could even be considered as a bug-fix, as this is how it should work if not all instances have the same number of repetitions.

[nikohansen]

see also #733, #745.

In the very first step, we should be able to read and use data where all instances are repeated exactly the same time.

[nikohansen]

FTR, when we use simulated restarts, the number of instances should not cause any troubles, because in this case the number of samples is fixed independently of the number of instances.

EDIT: see however #2042, and uniform sampling is still an issue.

[nikohansen]

A simple way to address the problem of different number of repetitions of instances would be to copy the data of underrepresented instances within the data set, such that all instances have the same number of repetitions. EDIT: the option cocopp.genericsettings.balance_instances triggers this behavior and is by default set to True.

[nikohansen]

I wonder about the current status? I believe we can read in any data but we do not normalize by per-instance repetition? See #1972. EDIT: balancing by data duplication can now be toggled via genericsettings.balance_instances

[nikohansen]

The new experimental script adds trials instead of within-trial restarts. The number of trials for each instance is the same. The postprocessing balances instance if their count is unbalanced.