Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running error #3

Open
Evelyn-coder opened this issue Jun 2, 2020 · 3 comments
Open

Running error #3

Evelyn-coder opened this issue Jun 2, 2020 · 3 comments

Comments

@Evelyn-coder
Copy link

Traceback (most recent call last):
File "train.py", line 172, in
main(args)
File "train.py", line 143, in main
train, eval, test, scaler = getattr(data.dataloader, dataset_setting['dataloader'])(settings)
File "/home/xinyilian/ST-MetaNet/flow-prediction/src/data/dataloader.py", line 109, in dataloader_all_sensors_seq2seq
return dataiter_all_sensors_seq2seq(train, scaler, setting),
File "/home/xinyilian/ST-MetaNet/flow-prediction/src/data/dataloader.py", line 90, in dataiter_all_sensors_seq2seq
feature = mx.nd.array(np.stack(feature)) # [B, N, D]
File "/home/xinyilian/.conda/envs/xyl/lib/python3.6/site-packages/mxnet/ndarray/utils.py", line 146, in array
return _array(source_array, ctx=ctx, dtype=dtype)
File "/home/xinyilian/.conda/envs/xyl/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 2505, in array
arr[:] = source_array
File "/home/xinyilian/.conda/envs/xyl/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 449, in setitem
self._set_nd_basic_indexing(key, value)
File "/home/xinyilian/.conda/envs/xyl/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 715, in _set_nd_basic_indexing
self._sync_copyfrom(value)
File "/home/xinyilian/.conda/envs/xyl/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 881, in _sync_copyfrom
ctypes.c_size_t(source_array.size)))
File "/home/xinyilian/.conda/envs/xyl/lib/python3.6/site-packages/mxnet/base.py", line 253, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [19:21:13] src/ndarray/ndarray_function.cc:51: Check failed: size == to->Size() (-2000107520 vs. 2294859776) : copying size mismatch, from: 18446744065709121536 bytes, to: 9179439104 bytes.
Stack trace:
[bt] (0) /home/xinyilian/.conda/envs/xyl/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x4b04cb) [0x7feb9aa234cb]
[bt] (1) /home/xinyilian/.conda/envs/xyl/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x281c85b) [0x7feb9cd8f85b]
[bt] (2) /home/xinyilian/.conda/envs/xyl/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::NDArray::SyncCopyFromCPU(void const*, unsigned long) const+0x27c) [0x7feb9cd1b59c]
[bt] (3) /home/xinyilian/.conda/envs/xyl/lib/python3.6/site-packages/mxnet/libmxnet.so(MXNDArraySyncCopyFromCPU+0x2b) [0x7feb9ca9790b]
[bt] (4) /home/xinyilian/.conda/envs/xyl/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7fec1d551ec0]
[bt] (5) /home/xinyilian/.conda/envs/xyl/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call+0x22d) [0x7fec1d55187d]
[bt] (6) /home/xinyilian/.conda/envs/xyl/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2ce) [0x7fec1d767ede]
[bt] (7) /home/xinyilian/.conda/envs/xyl/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(+0x13915) [0x7fec1d768915]
[bt] (8) python(_PyObject_FastCallDict+0x8b) [0x560bdc88ae3b]

@panzheyi
Copy link
Owner

panzheyi commented Jun 3, 2020

It seems that your mxnet version can only create an array with 2^32 bits of memory, causing the overflow.
The backend of mxnet often changes, so unexpected errors sometimes raised.
I suggest you to use the versions of packages specified in requirements.txt.

@Evelyn-coder
Copy link
Author

It seems that your mxnet version can only create an array with 2^32 bits of memory, causing the overflow.
The backend of mxnet often changes, so unexpected errors sometimes raised.
I suggest you to use the versions of packages specified in requirements.txt.

Thank you very much. You are really nice.
Finally I solved it by installing the MXnet with right gpu version. Thanks a lot.

@wanzhixiao
Copy link

It seems that your mxnet version can only create an array with 2^32 bits of memory, causing the overflow.
The backend of mxnet often changes, so unexpected errors sometimes raised.
I suggest you to use the versions of packages specified in requirements.txt.

Thank you very much. You are really nice.
Finally I solved it by installing the MXnet with right gpu version. Thanks a lot.

Hi,how do you installed mxnet-cu90 == 1.5.0b20190108? could you give a command?
when i install mxnet-cu90 == 1.5.0b20190108, there is no such version

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants