-
Notifications
You must be signed in to change notification settings - Fork 748
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
calculate_seasonal_error bug for multivariate data #2494
Comments
@gorold I believe the issue is that the layout of the forecast array should be different: gluonts/src/gluonts/model/forecast.py Lines 745 to 761 in 3231271
Still, it seems like even swapping the dimensions in your MWE results in shape issues both on the
@gorold if you don't mind the question: did you uncover this while working on your own quantile-regression based multivariate model? Because I don't think there are any in GluonTS as of now |
@lostella Thanks for getting back, the issue doesn't seem to arise from the QuantileForecast object. The dependency on forecasts in Regarding the dimensions of the QuantileForecast object, I was following this. Seems like the convention changes from v0.11.x to dev branch. gluonts/src/gluonts/model/forecast.py Lines 749 to 760 in d38ee93
I was combining multiple univariate quantile forecasters on a multivariate time series and using the MultivariateEvaluator to aggregate the metrics over the dimensions. |
In fact, one more thing is that while MASE and MSIS may not have been defined for multivariate time series, the straightforward extension would be to calculate the seasonal error independently across dimensions. So the gluonts/src/gluonts/evaluation/metrics.py Line 52 in cac5e6b
|
Sorry, I just realised that gluonts/src/gluonts/evaluation/_base.py Lines 720 to 723 in 8c29bca
Perhaps an assertion can be made in the get_metrics_per_ts function that inputs are univariate.
There is still an error in 0.11.x with multivariate from datetime import timedelta
import pandas as pd
import numpy as np
from gluonts.evaluation import MultivariateEvaluator
from gluonts.model.forecast import SampleForecast, QuantileForecast
from itertools import tee
dim = 5
time = 20
past = 40
quantile_forecast = QuantileForecast(
forecast_arrays=np.random.randn(3, dim, time), # following 0.11.x convention
forecast_keys=['mean', '0.95', '0.05'],
start_date=pd.to_datetime('today').to_period('H')
)
sample_forecast = SampleForecast(
samples=np.random.randn(100, time, dim),
start_date=pd.to_datetime('today').to_period('H')
)
time_series = pd.DataFrame(zip(*[np.arange(past + time) for _ in range(dim)]),
pd.period_range(start=pd.to_datetime('today') - timedelta(hours=past), periods=past+time, freq='H'))
evaluator = MultivariateEvaluator(seasonality=1)
# ok
sample_agg, sample_item = evaluator([time_series], [sample_forecast])
# not ok
quantile_agg, quantile_item = evaluator([time_series], [quantile_forecast]) Raises error:
|
Thanks for the further inspection. I agree, that would be the fix here |
Description
calculate_seasonal_error
is buggy when the inputs are multivariate data.past_data = extract_past_data(time_series, forecast)
results in(dim, time)
shaped array rather than a(time,)
shaped array, which is then passed intocalculate_seasonal_error
, which calls the follow code snippet.gluonts/src/gluonts/evaluation/_base.py
Lines 323 to 326 in cac5e6b
Rather than calculating the seasonal error, it is differencing between multivariate dimensions.
gluonts/src/gluonts/evaluation/metrics.py
Lines 49 to 50 in 4fef7e2
To Reproduce
(Please provide minimal example of code snippet that reproduces the error. For existing examples, please provide link.)
Error message or code output
'seasonal_error': 0.0
Desired message or code output
'seasonal_error': 1.0
Environment
The text was updated successfully, but these errors were encountered: