Deterministic algorithm benchmark

[donglinjy]

Why:
Some algorithm like gridsearch is deterministic when the space is defined, so it make no sense to same task multiple time for the average purpose.

Fix:
For the input algorithms of benchmark,
get_or_create_benchmark(..., algorithms=[...], ...)

Instead of keeping it exactly the same as experiment input
algorithms=["random", {"gridsearch": {"n_values": 50}}]

We change it as below so when some algorithm is deterministic, put True here.

algorithms = [
            {"algorithm": "random"},
            {"algorithm": {"gridsearch": {"n_values": 50}}, "deterministic": True},
        ]

[bouthilx]

The main problem for having deterministic defined in the algorithm class is that Assessment/Study are not working on the instantiated algorithms so they cannot access it, is that it? If we could access at least the algo class based on the algorithm name we could access the class.deterministic property.

[donglinjy]

The main problem for having deterministic defined in the algorithm class is that Assessment/Study are not working on the instantiated algorithms so they cannot access it, is that it? If we could access at least the algo class based on the algorithm name we could access the class.deterministic property.

Correct, current algorithm interfaces is asking for an instance through the factory first, or it is just a string. OptimizationAlgorithm.types and OptimizationAlgorithm.typenames could be a way to access the class object for a string algorithm name.

Although, algorithm level deterministic may not be always accurate, a random seed could make random algorithm reproducible actually.

[bouthilx]

Although, algorithm level deterministic may not be always accurate, a random seed could make random algorithm reproducible actually.

Good point! I did not think if this at all. So maybe defining if an algorithm is deterministic in the object itself is slightly more complicated.

Perhaps your solution is the simplest one. There is one issue however with supporting str and dict. We should support the dict in the form of {'algo_name': config} as well as {'algorithms': {'algo_name': config}, 'deterministic': True}.

[donglinjy]

@bouthilx we now support all the algorithms input format.

• ["random"]
• ["random" :{"seed": 1}]
• [{"algorithm": "random"}]
• [{"algorithm": {"random" :{"seed": 1}}}]
• [{"algorithm": "random", "deterministic": True}]

[bouthilx]

Alright! There is only few minor things left and then it is ready for merge. Thanks!

[bouthilx]

LGTM, thanks! :)