Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose NBEATSEnsemble aggregation_method and wrong dimensions #2592

Closed
gorold opened this issue Jan 25, 2023 · 1 comment · Fixed by #2598
Closed

Expose NBEATSEnsemble aggregation_method and wrong dimensions #2592

gorold opened this issue Jan 25, 2023 · 1 comment · Fixed by #2598
Labels
bug Something isn't working

Comments

@gorold
Copy link
Contributor

gorold commented Jan 25, 2023

Description

(A clear and concise description of what the bug is.)

  1. NBEATSEnsemblePredictor has the option of selecting aggregation_method, but it is not exposed to the estimator class. This means that there is no natural way of selecting this parameter. It is possible to interpret the forecasts from the ensemble as samples, and calculate the quantiles from an ensemble. This parameter should be exposed to the estimator class.
  2. Manually selecting the aggregation_method="none" still leads to a bug, wrong dimension size for SampleForecast. The bug fails silently when evaluated with the Evaluator class, and leads to erroneous metrics.

for item in dataset:
output = []
start_date = None
for iterator in iterators:
prediction = next(iterator)
# on order to avoid mypys complaints
assert isinstance(prediction, SampleForecast)
output.append(prediction.samples)
# get the forecast start date
if start_date is None:
start_date = prediction.start_date
output = np.stack(output, axis=0)
# aggregating output of different models
# default according to paper is median,
# but we can also make use of not aggregating
if self.aggregation_method == "median":
output = np.median(output, axis=0)
elif self.aggregation_method == "mean":
output = np.mean(output, axis=0)
else: # "none": do not aggregate
pass

Issue should be from here, where output is shape (N, 1, T)

To Reproduce

(Please provide minimal example of code snippet that reproduces the error. For existing examples, please provide link.)

from gluonts.mx import NBEATSEnsembleEstimator, NBEATSEnsemblePredictor
from gluonts.dataset.repository.datasets import get_dataset

dataset = get_dataset("airpassengers")

training_data, test_gen = split(dataset.train, offset=-36)
model = NBEATSEnsembleEstimator(
    prediction_length=12, freq="M", 
    meta_context_length=[2], meta_bagging_size=1,
    trainer=Trainer(epochs=1)
).train(training_data)

predictor = NBEATSEnsemblePredictor(12, model.predictors, "none")
test_data = test_gen.generate_instances(12)
forecast_it = predictor.predict(test_data.input)
forecast = next(forecast_it)

print(forecast.samples.shape)

Error message or code output

(Paste the complete error message, including stack trace, or the undesired output that the above snippet produces.)

(3, 1, 12)

Desired dimensionality should be (3, 12)

Environment

  • Operating system:
  • Python version:
  • GluonTS version: 0.11.6
  • MXNet version:

(Add as much information about your environment as possible, e.g. dependencies versions.)

@lostella
Copy link
Contributor

@gorold both points make perfect sense, thank you. A fix is ready in #2598, do you want to take a look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants