Getting optimal parameters after a study is complete

[vinaykakade]

Is there a recommended way of getting the optimal parameters when a study is complete? For example, it would be useful to get the parameters that have the maximum (or minimum) value for a the objective metric once the study is complete, so that the model could be further trained with those parameters.

[jlewi]

@YujiOshima @gaocegege Do you know the answer?

[jlewi]

@YujiOshima ping?

[YujiOshima]

@vinaykakade Do you mean retrain or fine-tuning?
If you mean retrain, the Hyperparameters are needed.
It is store Katib DB and you can find in KatibUI or get Katib API.

For fine-tuning, the trained weight parameters are needed.
It is not managed in Katib now.
It is exactly the experimental tracking area. (discussed here kubeflow/community#195 )
Katib will integrate Model management(StudioML or MLFlow) and the model data will be managed there.

[vinaykakade]

I meant for retraining. We are currently iterating on the study results and determine the winning combination, but it would be nice to support this in katib itself so that each of katib's customers don't need to implement the same functionality.

[YujiOshima]

@vinaykakade I agree. Easy to retraining is very useful.
How about introducing a one-shot job API.
We can specify a Trial ID or parameters directly, and it will be evaluated without any suggestion service.

[hougangliu]

I think #312 solved this requirement.
After studyJob completed, user can get bestTrialId from studyJob.Status.BestTrialID. Then call grpc api GetTrial to get trial detail which includes optimal parameters. For example:

    with grpc.insecure_channel('localhost:6790') as channel:
        stub = api_pb2_grpc.ManagerStub(channel)
        response = stub.GetTrial(api_pb2.GetTrialRequest(trial_id="n51572c70f020c3d"))

[YujiOshima]

@hougangliu I think it is not enough. We need to add a one-shot job API for specific trial ID. I will work on it.
/assign @YujiOshima

[richardsliu]

We can close this one since the design for v1alpha2 API covers this. Implementation will be completed in 0.6.

/close

[k8s-ci-robot]

@richardsliu: Closing this issue.

In response to this:

We can close this one since the design for v1alpha2 API covers this. Implementation will be completed in 0.6.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.