Noisy objective function not taken into account in SimpleExperiment when suggesting best parameters

[LukeAI]

I've been doing ax hyperparameter optimisation for a DNN doing regression on images like this:

    exp = SimpleExperiment(
        name=EXPERIMENT_NAME,
        search_space=dnn_search_space,
        evaluation_function=train_cross,
        objective_name="regression_error",
        minimize=True,
    )

    # sobol sample search space
    for i in range(20):
        exp.new_trial(generator_run=sobol.gen(1))

    # converge on best hyperparams
    best_arm = None
    for i in range(50):
        gpei = Models.GPEI(experiment=exp, data=exp.eval())
        generator_run = gpei.gen(1)
        best_arm, _ = generator_run.best_arm_predictions
        exp.new_trial(generator_run=generator_run)
        best_parameters = best_arm.parameters
        print(str(i) + " best params " + str(best_parameters))

and I have found that the "best parameters" recommended by ax tend to not change very much. This suggests that ax is giving me the hyperparameters that were evaluated and found to give the best result.

The problem with this is that the best results tend to be flukes - the training process is of course noisy and non-determinate and things that make the process more stochastic, such as very high learning rates and small batch sizes, tend to give more varied results. The more varied results will happen to include the best and worst results and on average, be worse than smoother more stable parameter sets. But Ax seems to just take the best result it finds and recommend this.

Is there some way of using Ax in which it will assume a noisy underlying object function and recommend the best hyperparameters based on an interpolation which uses all of the information available to it. rather than just, which trial scored best, one time?

[stevemandala]

Hey @LukeAI, thanks for raising this. I believe this was caused by a bug in simple experiment assuming 0.0 SEM. We recently pushed a fix on master, which should ensure we don't default to noise-less modeling when SEM isn't provided: f6ccdd7

[LukeAI]

@stevemandala ok, thanks for reply!
In the meantime how can I set SEM? It doesn't appear to be in the constructor

[lena-kashtelyan]

@LukeAI, you'd have to apply this change: f6ccdd7#diff-58c442e1539c8eedb46f78c90254cd976fb7462c0413543fa1c402cd5c6d5f3bR199-R201 to the SimpleExperiment code.

[LukeAI]

hmm. ok... do you know when the patch will come through on pip? Or if there is some other workaround?

[lena-kashtelyan]

@LukeAI, the patch should be part of the new stable version (and therefore on pip) within the next two weeks. In the meantime, you could install Ax master like this if you wanted: https://github.com/facebook/Ax#latest-version.

A good alternative would be to just not use SimpleExperiment, as it is intended for deprecation in favor of the Service API in the near future in any case (tutorial: https://ax.dev/tutorials/gpei_hartmann_service.html). If you wanted to have some control over the generation strategy used by the AxClient (so you can set custom number of trials to generate from Sobol, for instance), check out #199 for how to configure a custom generation strategy and pass it to AxClient.

[lena-kashtelyan]

This should now be fixed in latest stable release, 0.1.20.

[LukeAI]

@lena-kashtelyan I have upgraded to 0.1.20 and ran the same code, as above - but I am observing the same behaviour - no change in "best parameters" over time.

Maybe I have misunderstood how ax works? - I would expect each trial ran to be a an addition of new information which would change the recommended hyperparameters, at least a tiny bit.

Since this doesn't happen - I guess I am just seeing the specific hyperparams that gave the best results in one trial, rather than an interpolation based on all available information.

Is this correct? If so, is this intentional/expected behaviour? Or am I using ax incorrectly?

[ldworkin]

@LukeAI, is it possible for you to get us a reproducible example?