HPO logger handler #3061
HPO logger handler #3061
Conversation
|
|
||
| from ignite.engine import Engine, Events | ||
|
|
||
| _DEFAULT_METRIC_PATH = '/tmp/hypertune/output.metrics' |
There was a problem hiding this comment.
I wonder whether this code could run on windows. If yes, we may need to provide OS-agnostic path using python tempfile module.
There was a problem hiding this comment.
this is the path where the listener of the GCP HPO orchestration has a look. Whatever is dumped here (in terms of metric values) becomes visible to the Vertex platform. I found this path here https://github.com/GoogleCloudPlatform/cloudml-hypertune/blob/8e3530e1c4926ac64cf28330d1104b838e07a468/hypertune/hypertune.py#L24
Should we keep it?
There was a problem hiding this comment.
Thanks for the PR @St3V0Bay !
Can you please run this script to reformat the code to our format:
bash ./tests/run_code_style.sh install
bash ./tests/run_code_style.sh fmt
and let's also run mypy:
bash ./tests/run_code_style.sh mypy
| def __init__( | ||
| self, | ||
| evaluator: Engine | None = None, | ||
| metric_tag: UserString = 'training/hptuning/metric' |
There was a problem hiding this comment.
Can't we use str here ?
| metric_tag: UserString = 'training/hptuning/metric' | |
| metric_tag: str = 'training/hptuning/metric' |
| Args: | ||
| engine: Ignite Engine, it can be a trainer, validator or evaluator. | ||
| """ | ||
| with self.lock: |
There was a problem hiding this comment.
why do we need a lock here ?
| self.metrics.clear() | ||
| self.metrics.update(state_dict[MetricLoggerKeys.METRICS]) | ||
|
|
||
| hpologger = HPOLogger() |
There was a problem hiding this comment.
I think we can remove this line.
|
|
||
| class HPOLogger(object): | ||
| """ | ||
| Makes selected metric accessible for use by GCP Vertex AI hyperparameter tuning jobs. By adding only this |
There was a problem hiding this comment.
Can you please add a link to GCP Vertex AI as
`GCP Vertex AI <link>`_
|
|
||
| Args: | ||
| evaluator: Evaluator to consume metric results from at the end of its evaluation run | ||
| metric_tag: Converts the metric value coming from the trainer/evaluator's state into a storable value |
There was a problem hiding this comment.
Below this line we need to add .. versionadded:: 0.5.0 tag.
See https://github.com/pytorch/ignite/blob/master/CONTRIBUTING.md#writing-documentation for details
Fixes #3023
Description:
Check list: