-
Notifications
You must be signed in to change notification settings - Fork 744
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rework ev. #2914
Rework ev. #2914
Conversation
Do we need 3 "layers" for this: |
I think we need it. Otherwise we loose the flexibility that we wanted in the first place. |
src/gluonts/ev/metrics.py
Outdated
A derived metric updates multiple, simpler metrics independently and in | ||
the end combines their results as defined in `post_process`.""" | ||
|
||
evaluators: Dict[str, Metric] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would metrics
be a better name, to avoid confusion?
src/gluonts/ev/metrics.py
Outdated
def update_all(self, stream: Iterator[Mapping[str, np.ndarray]]) -> None: | ||
for element in stream: | ||
self.update(element) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This could return self
maybe
src/gluonts/ev/__init__.py
Outdated
def evaluate(metrics, data_batches, axis=None): | ||
evaluator = metrics(axis) | ||
evaluator.update_all(data_batches) | ||
return evaluator.get() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If update_all
returned self
, this would be metrics(axis).update_all(data_batches).get()
src/gluonts/ev/metrics.py
Outdated
def update(self, data: Mapping[str, np.ndarray]) -> None: | ||
for metric in self.metrics: | ||
metric.update(data) | ||
|
||
def update_all(self, stream: Iterator[Mapping[str, np.ndarray]]) -> None: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These methods could use a docstring I guess
@@ -34,115 +43,250 @@ | |||
) | |||
|
|||
|
|||
@dataclass | |||
class MetricCollection: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could this have Metric
as base class?
Changes:
RMSE
now isRMSE[mean]
, indicating that we compare against the mean prediction.Evaluator
toMetric
, andMetric
toMetricDefinition
.MetricCollection
andMetricDefinitionCollection
to group respective objects.evaluate
function to run evaluations on metrics end to end.Usage:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Please tag this pr with at least one of these labels to make our release process faster: BREAKING, new feature, bug fix, other change, dev setup