-
Notifications
You must be signed in to change notification settings - Fork 777
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
doc(katib): update push-based metrics collector. #3844
base: master
Are you sure you want to change the base?
doc(katib): update push-based metrics collector. #3844
Conversation
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Thanks for adding this @Electronic-Waste! /assign @hbelmiro @StefanoFioravanzo |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left some explanations for the fields in tune
function. If you have any questions, don't hesitate to contact me.
content/en/docs/components/katib/user-guides/metrics-collector.md
Outdated
Show resolved
Hide resolved
content/en/docs/components/katib/user-guides/metrics-collector.md
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/lgtm
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for this. I left a few comments.
/assign @kubeflow/wg-training-leads
content/en/docs/components/katib/user-guides/metrics-collector.md
Outdated
Show resolved
Hide resolved
content/en/docs/components/katib/user-guides/metrics-collector.md
Outdated
Show resolved
Hide resolved
Use tune function and specify the `metrics_collector_config` field. You can reference to the following example: | ||
|
||
``` | ||
import kubeflow.katib as katib |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's use simple example here and remove all unnecessary parameters and function calls.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@andreyvelich Sorry, I may not fully understand what you mean "unnecessary parameters and function calls".
I copied the example in the get-started chapter and replaced print
with report_metrics()
. Do you mean that I don't need to call the tune()
function and just define a main function in the example?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think, we should just show example how user can use tune
API with push-based
metrics collector without any additional parameters they should set. For example:
import kubeflow.katib as katib
def objective(parameters):
import time
import kubeflow.katib as katib
time.sleep(5)
result = 4 * int(parameters["a"])
# Push metrics to Katib DB.
katib.report_metrics({"result": result})
katib.KatibClient().tune(
name="push-metrics-exp",
objective=objective,
parameters= {"a": katib.search.int(min=10, max=20)}
objective_metric_name="result",
max_trial_count=2,
metrics_collector_config={"kind": "Push"},
# When SDK is released, replace it with packages_to_install=["kubeflow-katib==0.18.0"]
packages_to_install=["git+https://github.com/kubeflow/katib.git@master#subdirectory=sdk/python/v1beta1"],
)
That should allow user to focus on important changes they need to make to try this out.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@andreyvelich Thanks for your clarification! I'll update the blog.
|
||
- [`tune`](https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/kubeflow/katib/api/katib_client.py#L166) function | ||
|
||
Use tune function and specify the `metrics_collector_config` field. You can reference to the following example: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You might need to explain how report_metrics()
functions works separately. E.g. user needs to install kubeflow-katib
SDK in their environment, and the report_metrics()
will automatically add the timestamp for metrics.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SGTM. I'll add the explanation to report_metrics()
.
New changes are detected. LGTM label has been removed. |
Signed-off-by: Electronic-Waste <[email protected]>
Signed-off-by: Electronic-Waste <[email protected]>
05975ad
to
0088100
Compare
@kubeflow/wg-automl-leads I've made some changes to the PR as you request. PTAL👀 if you have time. |
This PR aims at guiding users using push-based metrics collection: kubeflow/katib#2340.
I rewrote the metrics-collector chapter and divided it into three parts:
None
MC.Push
MC, with an example attached.cc👀 @kubeflow/wg-automl-leads . Please let me know if you have any suggestions.