Rework ev. #2914
Rework ev. #2914
Conversation
lostella
added
the
BREAKING
|
Do we need 3 "layers" for this: |
|
I think we need it. Otherwise we loose the flexibility that we wanted in the first place. |
src/gluonts/ev/metrics.py
Outdated
| A derived metric updates multiple, simpler metrics independently and in | ||
| the end combines their results as defined in `post_process`.""" | ||
|
|
||
| evaluators: Dict[str, Metric] |
There was a problem hiding this comment.
would metrics be a better name, to avoid confusion?
src/gluonts/ev/metrics.py
Outdated
| def update_all(self, stream: Iterator[Mapping[str, np.ndarray]]) -> None: | ||
| for element in stream: | ||
| self.update(element) |
There was a problem hiding this comment.
This could return self maybe
src/gluonts/ev/__init__.py
Outdated
| def evaluate(metrics, data_batches, axis=None): | ||
| evaluator = metrics(axis) | ||
| evaluator.update_all(data_batches) | ||
| return evaluator.get() |
There was a problem hiding this comment.
If update_all returned self, this would be metrics(axis).update_all(data_batches).get()
src/gluonts/ev/metrics.py
Outdated
| def update(self, data: Mapping[str, np.ndarray]) -> None: | ||
| for metric in self.metrics: | ||
| metric.update(data) | ||
|
|
||
| def update_all(self, stream: Iterator[Mapping[str, np.ndarray]]) -> None: |
There was a problem hiding this comment.
These methods could use a docstring I guess
| @@ -34,115 +43,250 @@ | |||
| ) | |||
|
|
|||
|
|
|||
| @dataclass | |||
| class MetricCollection: | |||
There was a problem hiding this comment.
Could this have Metric as base class?
Changes:
RMSEnow isRMSE[mean], indicating that we compare against the mean prediction.EvaluatortoMetric, andMetrictoMetricDefinition.MetricCollectionandMetricDefinitionCollectionto group respective objects.evaluatefunction to run evaluations on metrics end to end.Usage:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Please tag this pr with at least one of these labels to make our release process faster: BREAKING, new feature, bug fix, other change, dev setup