Test for Bayesian Optimization Algo

[garganubhav]

I am working on this

[jlewi]

@garganubhav What is the current status of this? Can you provide more information about what you are planning to deliver and when?

For example, will the test just verify that the bayesian algorithm generates suggestions without errors, or will it also check correctness?

[garganubhav]

Currently, I am not working on this. I started but then the BO algo was not working in Katib.

[YujiOshima]

@garganubhav What is the current status of this? BO can work now.

[garganubhav]

@YujiOshima I don't plan to work on this in near future.

[jdplatt]

Can I get a few more details on what is required here? I'm not sure if you are looking for a set of unit tests to make sure the BO code runs correctly, or more of a benchmark on a standard ML tasks to compare the performance of BO with something simpler like random search. Either way I am happy to help out.

[johnugeorge]

Thanks @jdplatt This issue is to track unit tests for BO algo. See unit tests for other algos https://github.com/kubeflow/katib/tree/master/pkg/suggestion

[jdplatt]

I took a look at the code and have a few questions:

1. Do you have any existing unit tests for python code in the repo? I couldn’t find any but wanted to double check. If there are I’ll make sure to follow the existing workflow.
2. I have a few questions about the structure (e.g. why do each of the other algorithms get their own service while the BO service contains multiple algorithms) and what different parameters mean. Where is the best place to get answers to these sort of questions? I’ll try to add some docs to the code along side the tests.

[johnugeorge]

/cc @YujiOshima

[YujiOshima]

@jdplatt Thank you!
Unfortunately, we don't have a test with python in the repo. We want a function level unit test for BO. Please use some libraries like unittest.
The algorithm level test will be defined here. This code will call API and test end to end algorithms. It is an advanced task.

why do each of the other algorithms get their own service while the BO service contains multiple algorithms

You are right. It would be better to split them.

I’ll try to add some docs to the code along side the tests.

Great! Please make a new issue for your questions.

[jlewi]

@YujiOshima any update on this?

[jdplatt]

@jlewi This got merged in #406

[YujiOshima]

@jdplatt Thanks!
/close

[k8s-ci-robot]

@YujiOshima: Closing this issue.

In response to this:

@jdplatt Thanks!
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.