[Bug report] Logger logging noisy function values instead of noisy-free ones for the bbob-noisy suite (?).

[lorenzo-consoli]

Bug description

I ran some experiments on the bbob-noisy suite and yesterday, while we were discussing about the results with @nikohansen and @brockho,, we found out a pretty weird behavior of the algorithms on the newly implemented bbob-noisy test suite, thus we decided to run a random search as well and compare the results of our run with the data archive from 2009 and we found out that they seemed to be pretty different, so we decided to look at the exdata folder, which is created (I gues by the cma code) when an experiment is run and we found out that, most likely, the root cause of this weird behaviour, except for some difference in the algorithm's settings (which are not relevant in the case of random search), were probably due to some problem in the logging facilities, especially relative to the logging of the noisy objective functions. Indeed, if we look at one example, let' say run on the F101 I 1 of the bbob-noisy suite, and wze run head ./exdata/results/fmin2_1000x_on_bbob/data_f1/bbobexp_f1_DIM2.dat, the first lines are:

% f evaluations | g evaluations | best noise-free fitness - Fopt (7.948000000000e+01) + sum g_i+ | measured fitness | best measured fitness or single-digit g-values | x1 | x2...
1 0 +1.402094080e+00 +8.088209408e+01 +8.088209408e+01 +0.0000e+00 +0.0000e+00

So, if the third number (+1.402094080e+00) is the noise free function value minus fopt (7.948000000000e+01) and the fourth is the measured (noisy) fitness (+8.088209408e+01), the we would expect, since the function is noisy, to have:

noise-free fitness - Fopt (7.948000000000e+01) + sum g_i + Fopt != measured fitness , but

1.402094080e+00 + 7.948000000000e+01 = +8.088209408e+01

So this means that the logger is actually logging the noisy function values, which I guess is not the wanted behavior and is probably going to reveal itself as being the cause of these weird results that we were observing yesterday.

Does what I am saying make any sense at all??

Further Considerations

We though that a potentially useful way to proceed would be to add a flag to the coco_problem_t "class", something like int is_noisy_problem and then, at each evaluation, saving the noise free function value somewhere (most likely in the data attribute of coco_problem_t) and then have the logger check the is_noisy_problem flag for each problem and go and retrieve this noise free value that we will be storing somewhere instead of the function value in the y array...

I will now create a new branch from feat-noisy-suite and try to adjust the code accordingly.

Thank you very much for your time and attention!

Best regards,

Lorenzo

[lorenzo-consoli]

Actually, I was just going through the logger_bbob.c file and I found the logger_bbob_evaluate function which seems to be taking the problem -> inner_problem attribute for evaluating the solution... This makes it even more weirder as I think it would be supposed to log the noise free value in this way, if I am not mistaken

[nikohansen]

Maybe you can link the to place in the code you are referring to, but isn't the inner_problem the "actual problem" (which is noisy) while the (outer) problem is the "observed" problem wrapped by the logger/observer?

[lorenzo-consoli]

isn't the inner_problem the "actual problem" (which is noisy) while the (outer) problem is the "observed" problem wrapped by the logger/observer

To me it actually looks like it is all the way round, here's why:

In the code-experiments/src/logger_bbob.c file, at line 364 (on the development branch), there is the definition of a function with the following signature:

static void logger_bbob_evaluate(coco_problem_t *problem, const double *x, double *y) 

I assume it is the function that the logger calls for evaluating the objective and constraints (when present).

The evaluation of the objective function, is done in the following way:

static void logger_bbob_evaluate(coco_problem_t *problem, const double *x, double *y) {
  size_t i;
  double y_logged, max_value = 0, sum_constraints;
  double *constraints = NULL;
  logger_bbob_data_t *logger = (logger_bbob_data_t *) coco_problem_transformed_get_data(problem);
  coco_problem_t *inner_problem = coco_problem_transformed_get_inner_problem(problem);
  const int is_feasible = problem->number_of_constraints <= 0 || coco_is_feasible(inner_problem, x, NULL);

  coco_debug("Started logger_bbob_evaluate()");

  if (!logger->is_initialized) {
    logger_bbob_initialize(logger, problem->is_opt_known);
  }
  if ((coco_log_level >= COCO_DEBUG) && logger->num_func_evaluations == 0) {
    coco_debug("%4lu: ", (unsigned long) inner_problem->suite_dep_index);
    coco_debug("on problem %s ... ", coco_problem_get_id(inner_problem));
  }

  /* Fulfill contract of a COCO evaluate function */
  coco_evaluate_function(inner_problem, x, y);

 ... //more stuff that is not relevant for the problem
}

The only lines that are of interest for us now are the following two:

  coco_problem_t *inner_problem = coco_problem_transformed_get_inner_problem(problem);

and the last one that I pasted here:

  coco_evaluate_function(inner_problem, x, y);

So it looks like that the logger uses the problem -> inner_problem -> evaluate_function to evaluate the objective function and write the results on the y array.

If we look in the code-experiments/src/transform_obj_<noise-model>.c file, as well as in the others, the allocation of the coco_problem_t happens by means of the coco_problem_transformed_allocate function, defined in code-experiments/src/coco_problem.c and the argument passed to this function is the noise-free problem (or, more generally, the coco_problem_t before a given transformations is applied to it).
This argument is then set as the inner_problem attribute of the newly created (transformed/noisy) coco_problem_t.
To the transformed problem is then passed the transformed_obj_<noise-model>_evaluate_function as its evaluate_function attribute (otherwise it would still evaluate the noise-free problem).

It is defined in the following way:

                                                         void *user_data,
                                                         coco_data_free_function_t data_free_function,
                                                         const char *name_prefix) {
  coco_problem_transformed_data_t *problem;
  coco_problem_t *inner_copy;
  char *old_name = coco_strdup(inner_problem->problem_name);

  problem = (coco_problem_transformed_data_t *) coco_allocate_memory(sizeof(*problem));
  problem->inner_problem = inner_problem;
  problem->data = user_data;
  problem->data_free_function = data_free_function;

  inner_copy = coco_problem_duplicate(inner_problem);
  inner_copy->evaluate_function = coco_problem_transformed_evaluate_function;
  inner_copy->evaluate_constraint = coco_problem_transformed_evaluate_constraint;
  inner_copy->evaluate_gradient = bbob_problem_transformed_evaluate_gradient;
  inner_copy->recommend_solution = coco_problem_transformed_recommend_solution;
  inner_copy->problem_free_function = coco_problem_transformed_free;
  inner_copy->data = problem;

  coco_problem_set_name(inner_copy, "%s(%s)", name_prefix, old_name);
  coco_free_memory(old_name);

  return inner_copy;
}

I think that this should mean that the logger is still logging the correct 'noise-free' value...

If this is actually the case, for now I wouldn't be able to say what might have cause the strange behavior we observed the other day by the random search.

[nikohansen]

A technical hint, you can create links to the code by clicking on the line number when looking at the code at Github in the browser.

[nikohansen]

I don't see how this considers my original point though. I think, as I wrote, the problem is "observed" by "wrapping the logger/observer around the problem" in the same way as other transformations are wrapped. I suspect this code is used when a problem is logged like here, am I wrong?

Given that by evaluating the inner_problem, the logger dismisses the last applied transformation in the noisy case (that is, the noise), I would not understand what happens in the noisefree case where this transformation may be relevant to the problem?