Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

SSD example CUDA error: too many resources requested for launch #5170

Closed
ysh329 opened this issue Feb 28, 2017 · 2 comments
Closed

SSD example CUDA error: too many resources requested for launch #5170

ysh329 opened this issue Feb 28, 2017 · 2 comments

Comments

@ysh329
Copy link
Contributor

ysh329 commented Feb 28, 2017

It's okay to run CPU version SSD exmaple, but there's something wrong with CUDA. The main cuda error log is: example/ssd/operator/multibox_detection.cu:199: Check failed: error == cudaSuccess (7 vs. 0) too many resources requested for launch

Environment info

Operating System: TegraX1 Ubuntu 16.04.2 LTS
Linux tegra-ubuntu 3.10.96-tegra

Compiler: gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4)

Package used (Python/R/Scala/Julia):python

MXNet version:0.9.4

Or if installed from source: Yes

MXNet commit hash (git rev-parse HEAD):

If you are using python package, please provide

Python version and distribution: Python 2.7.12

Error Message:

Please paste the full error message, including stack trace.
(py-mxnet-SSD) yuanshuai@tegra-ubuntu:~/sdcard/code/mxnet_SSD_cuda_openblas/example/ssd$ python demo.py --epoch 0 --images ./data/demo/dog.jpg --thresh 0.5
[09:58:45] src/nnvm/legacy_json_util.cc:175: Loading symbol saved by previous version v0.8.0. Attempting to upgrade...
[09:58:53] /mnt/sdcard/yuanshuai/code/mxnet_SSD_cuda_openblas/dmlc-core/include/dmlc/./logging.h:300: [09:58:53] example/ssd/operator/multibox_detection.cu:199: Check failed: error == cudaSuccess (7 vs. 0) too many resources requested for launch

Stack trace returned 7 entries:
[bt] (0) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x48) [0x7f737f2da0]
[bt] (1) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(ZN5mxnet2op19MultiBoxDetectionOpIN7mshadow3gpuEfE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESD_SD+0x1b18) [0x7f74a34a40]
[bt] (2) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(+0xf2a048) [0x7f740f5048]
[bt] (3) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x88) [0x7f74095b68]
[bt] (4) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x88) [0x7f7409c6c0]
[bt] (5) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f66e24280]
[bt] (6) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7f7e02dfc4]

[09:58:53] /mnt/sdcard/yuanshuai/code/mxnet_SSD_cuda_openblas/dmlc-core/include/dmlc/./logging.h:300: [09:58:53] src/engine/./threaded_engine.h:336: [09:58:53] example/ssd/operator/multibox_detection.cu:199: Check failed: error == cudaSuccess (7 vs. 0) too many resources requested for launch

Stack trace returned 7 entries:
[bt] (0) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x48) [0x7f737f2da0]
[bt] (1) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(ZN5mxnet2op19MultiBoxDetectionOpIN7mshadow3gpuEfE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESD_SD+0x1b18) [0x7f74a34a40]
[bt] (2) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(+0xf2a048) [0x7f740f5048]
[bt] (3) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x88) [0x7f74095b68]
[bt] (4) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x88) [0x7f7409c6c0]
[bt] (5) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f66e24280]
[bt] (6) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7f7e02dfc4]

An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 5 entries:
[bt] (0) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x48) [0x7f737f2da0]
[bt] (1) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x2d0) [0x7f74095db0]
[bt] (2) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x88) [0x7f7409c6c0]
[bt] (3) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f66e24280]
[bt] (4) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7f7e02dfc4]

terminate called after throwing an instance of 'dmlc::Error'
what(): [09:58:53] src/engine/./threaded_engine.h:336: [09:58:53] example/ssd/operator/multibox_detection.cu:199: Check failed: error == cudaSuccess (7 vs. 0) too many resources requested for launch

Stack trace returned 7 entries:
[bt] (0) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x48) [0x7f737f2da0]
[bt] (1) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(ZN5mxnet2op19MultiBoxDetectionOpIN7mshadow3gpuEfE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESD_SD+0x1b18) [0x7f74a34a40]
[bt] (2) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(+0xf2a048) [0x7f740f5048]
[bt] (3) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x88) [0x7f74095b68]
[bt] (4) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x88) [0x7f7409c6c0]
[bt] (5) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f66e24280]
[bt] (6) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7f7e02dfc4]

An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 5 entries:
[bt] (0) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x48) [0x7f737f2da0]
[bt] (1) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x2d0) [0x7f74095db0]
[bt] (2) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x88) [0x7f7409c6c0]
[bt] (3) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f66e24280]
[bt] (4) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7f7e02dfc4]

Aborted
@ysh329 ysh329 changed the title SSD example CUDA error SSD example CUDA error: too many resources requested for launch Feb 28, 2017
@ysh329
Copy link
Contributor Author

ysh329 commented Feb 28, 2017

I found there may be something wrong with mxnet-ssd code version.

I fixed it after I clone the original mxnet-ssd contributor's repo from his own code repo(https://github.com/zhreshold/mxnet-ssd) and then tried as its README says, finally it worked!

@ysh329 ysh329 closed this as completed Feb 28, 2017
@ysh329
Copy link
Contributor Author

ysh329 commented Feb 28, 2017 •
edited
Loading

Before I fixed it, I referred this issue titled src/operator/multibox_detection.cu:198: Check failed: (error) == (cudaSuccess) too many resources requested for launch · Issue #13 · zhreshold/mxnet-ssd, however it didn't work for me. Because there is no make files in official ./mxnet/examples/ssd dir, I couldn't compile the modified cuda codes.

@ysh329 ysh329 mentioned this issue Mar 14, 2017
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ysh329