MXNetError: vector<T> too long" occurred in the official tutorial

[AIAficionado]

Description

Hi All,
So I have recently installed Gluon-TS on Python 3.7.3 with the following PC:

OS: Windows 7
CPU: Intel(R) Xeon(R) E3-1505M v5 @2.80Ghz
RAM: 32 GB
System Type: 64 bit OS

Python: 3.7.3
pandas: 0.24.2
numpy: 1.14.6
mxnet: 1.4.1
gluonts: 0.4.2
jupyter: 1.0.0
jupyterlab: 1.2.3
notebook: 6.0.2

To Reproduce

for test_entry, forecast in zip(test_data, predictor.predict(test_data)):
to_pandas(test_entry)[-60:].plot(linewidth=2)
forecast.plot(color='g', prediction_intervals=[50.0, 90.0])
plt.grid(which='both')

tutorial

Error message or code output

---

MXNetError Traceback (most recent call last)
in
6 from gluonts.dataset.util import to_pandas
7
----> 8 for test_entry, forecast in zip(test_data, predictor.predict(test_data)):
9 to_pandas(test_entry)[-60:].plot(linewidth=2)
10 forecast.plot(color='g', prediction_intervals=[50.0, 90.0])

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\gluonts\model\predictor.py in predict(self, dataset, num_samples)
307 freq=self.freq,
308 output_transform=self.output_transform,
--> 309 num_samples=num_samples,
310 )
311

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\gluonts\model\forecast_generator.py in call(self, inference_data_loader, prediction_net, input_names, freq, output_transform, num_samples, **kwargs)
195 for batch in inference_data_loader:
196 inputs = [batch[k] for k in input_names]
--> 197 outputs = prediction_net(*inputs).asnumpy()
198 if output_transform is not None:
199 outputs = output_transform(batch, outputs)

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\mxnet\gluon\block.py in call(self, *args)
546
547 def forward(self, *args):
--> 548 """Overrides to implement forward computation using :py:class:NDArray. Only
549 accepts positional arguments.
550

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\mxnet\gluon\block.py in forward(self, x, *args)
923 with self.name_scope():
924 return self.hybrid_forward(symbol, x, *args, **params)
--> 925
926 def hybrid_forward(self, F, x, *args, **kwargs):
927 """Overrides to construct symbolic graph for this Block.

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\gluonts\model\deepar_network.py in hybrid_forward(self, F, feat_static_cat, feat_static_real, past_time_feat, past_target, past_observed_values, future_time_feat)
603 static_feat=static_feat,
604 scale=scale,
--> 605 begin_states=state,
606 )

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\gluonts\model\deepar_network.py in sampling_decoder(self, F, static_feat, past_target, time_feat, scale, begin_states)
536
537 # (batch_size * num_samples, 1, *target_shape)
--> 538 new_samples = distr.sample(dtype=self.dtype)
539
540 # (batch_size * num_samples, seq_len, *target_shape)

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\gluonts\distribution\transformed_distribution.py in sample(self, num_samples, dtype)
87 with autograd.pause():
88 s = self.base_distribution.sample(
---> 89 num_samples=num_samples, dtype=dtype
90 )
91 for t in self.transforms:

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\gluonts\distribution\student_t.py in sample(self, num_samples, dtype)
116 sigma=self.sigma,
117 nu=self.nu,
--> 118 num_samples=num_samples,
119 )
120

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\gluonts\distribution\distribution.py in _sample_multiple(sample_func, num_samples, *args, **kwargs)
300 k: _expand_param(v, num_samples) for k, v in kwargs.items()
301 }
--> 302 samples = sample_func(*args_expanded, **kwargs_expanded)
303 return samples

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\gluonts\distribution\student_t.py in s(mu, sigma, nu)
104 F = self.F
105 gammas = F.sample_gamma(
--> 106 alpha=nu / 2.0, beta=2.0 / (nu * F.square(sigma)), dtype=dtype
107 )
108 normal = F.sample_normal(

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\mxnet\ndarray\register.py in sample_gamma(alpha, beta, shape, dtype, out, name, **kwargs)

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\mxnet_ctypes\ndarray.py in _imperative_invoke(handle, ndargs, keys, vals, out)
90 c_str_array(keys),
91 c_str_array([str(s) for s in vals]),
---> 92 ctypes.byref(out_stypes)))
93
94 if original_output is not None:

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\mxnet\base.py in check_call(ret)
251 if ret != 0:
252 raise MXNetError(py_str(_LIB.MXGetLastError()))
--> 253
254
255 if sys.version_info[0] < 3:

MXNetError: vector too long

Environment

• Operating system: Windows 7
• Python version: 3.7.3
• GluonTS version: 0.4.2

(Add as much information about your environment as possible, e.g. dependencies versions.)

Following this tutorial , however I still get the same error. It seems like the issue has been raised before here but I have the adequate version of MXNet for Gluon and I still get the error.

Can anyone please help?

[mbohlkeschneider]

Which specific version of Mxnet do you use @AIAficionado ?

Can you post GluonTs/Mxnet/Python versions here?

[AIAficionado]

Which specific version of Mxnet do you use @AIAficionado ?

Can you post GluonTs/Mxnet/Python versions here?

Sorry,
GluonTs - 0.4.2
MXNet - 1.4.1
Python - 3.7.3

[ehsanmok]

@AIAficionado could you please verify upgrading mxnet to 1.6.0 resolves this issue? suspecting it's related to this.

[AIAficionado]

@AIAficionado could you please verify upgrading mxnet to 1.6.0 resolves this issue? suspecting it's related to this.

Hi upgraded to MxNet to 1.6.0 but the traceback still persists:

---

MXNetError Traceback (most recent call last)
in
6 from gluonts.dataset.util import to_pandas
7
----> 8 for test_entry, forecast in zip(test_data, predictor.predict(test_data)):
9 to_pandas(test_entry)[-60:].plot(linewidth=2)
10 forecast.plot(color='g', prediction_intervals=[50.0, 90.0])

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\gluonts\model\predictor.py in predict(self, dataset, num_samples)
307 freq=self.freq,
308 output_transform=self.output_transform,
--> 309 num_samples=num_samples,
310 )
311

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\gluonts\model\forecast_generator.py in call(self, inference_data_loader, prediction_net, input_names, freq, output_transform, num_samples, **kwargs)
195 for batch in inference_data_loader:
196 inputs = [batch[k] for k in input_names]
--> 197 outputs = prediction_net(*inputs).asnumpy()
198 if output_transform is not None:
199 outputs = output_transform(batch, outputs)

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\mxnet\gluon\block.py in call(self, *args)
546 "Set allow_missing=True to ignore missing parameters."%(
547 name, filename, _brief_print_list(loaded.keys()))
--> 548 for name in loaded:
549 if not ignore_extra and name not in params:
550 raise ValueError(

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\mxnet\gluon\block.py in forward(self, x, *args)
923 params = {i: j.var() for i, j in self._reg_params.items()}
924 with self.name_scope():
--> 925 out = self.hybrid_forward(symbol, *grouped_inputs, **params) # pylint: disable=no-value-for-parameter
926 out, self._out_format = _flatten(out, "output")
927

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\gluonts\model\deepar_network.py in hybrid_forward(self, F, feat_static_cat, feat_static_real, past_time_feat, past_target, past_observed_values, future_time_feat)
603 static_feat=static_feat,
604 scale=scale,
--> 605 begin_states=state,
606 )

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\gluonts\model\deepar_network.py in sampling_decoder(self, F, static_feat, past_target, time_feat, scale, begin_states)
536
537 # (batch_size * num_samples, 1, *target_shape)
--> 538 new_samples = distr.sample(dtype=self.dtype)
539
540 # (batch_size * num_samples, seq_len, *target_shape)

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\gluonts\distribution\transformed_distribution.py in sample(self, num_samples, dtype)
87 with autograd.pause():
88 s = self.base_distribution.sample(
---> 89 num_samples=num_samples, dtype=dtype
90 )
91 for t in self.transforms:

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\gluonts\distribution\student_t.py in sample(self, num_samples, dtype)
116 sigma=self.sigma,
117 nu=self.nu,
--> 118 num_samples=num_samples,
119 )
120

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\gluonts\distribution\distribution.py in _sample_multiple(sample_func, num_samples, *args, **kwargs)
300 k: _expand_param(v, num_samples) for k, v in kwargs.items()
301 }
--> 302 samples = sample_func(*args_expanded, **kwargs_expanded)
303 return samples

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\gluonts\distribution\student_t.py in s(mu, sigma, nu)
104 F = self.F
105 gammas = F.sample_gamma(
--> 106 alpha=nu / 2.0, beta=2.0 / (nu * F.square(sigma)), dtype=dtype
107 )
108 normal = F.sample_normal(

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\mxnet\ndarray\register.py in sample_gamma(alpha, beta, shape, dtype, out, name, **kwargs)

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\mxnet_ctypes\ndarray.py in _imperative_invoke(handle, ndargs, keys, vals, out)
90 original_output = None
91 output_vars = ctypes.POINTER(NDArrayHandle)()
---> 92 num_output = ctypes.c_int(0)
93
94 # return output stypes to avoid the c_api call for checking

~\AppData\Local\Continuum\anaconda3\envs\myenv\lib\site-packages\mxnet\base.py in check_call(ret)
251 ret : int
252 return value from API calls.
--> 253 """
254 if ret != 0:
255 raise MXNetError(py_str(_LIB.MXGetLastError()))

MXNetError: vector too long

[ehsanmok]

@AIAficionado Thanks! it'd be great if you could make a Minimal Viable Example out of it and create an MXNet issue on windows since running the tutorial is fine in Ubuntu and Mac.

[LeapGamer]

I'm getting the same error. I am python 3.6, mxnet-cu92 1.4.1. I installed these versions after reading the previously linked thread. What is the current recommended version of everything for a cuda or cudamkl version working with gluon?

Edit: actually, i always comment too soon. I read my pip error messages more carefully and picked the last 1.6.0 build and did

pip install --upgrade mxnet-cu92==1.6.0b20191118 gluonts

Which solved my problem!

[lostella]

closing since this appears to have been caused by an old mxnet release