Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Segmentation fault during ab load test #6852

Closed
fasahath opened this issue Jun 28, 2017 · 1 comment
Closed

Segmentation fault during ab load test #6852

fasahath opened this issue Jun 28, 2017 · 1 comment

Comments

@fasahath
Copy link

fasahath commented Jun 28, 2017

Environment info

Operating System:
Ubuntu 16.04.2 LTS on AWS p2.xlarge

Compiler:
gcc version 5.4.0

Package used (Python/R/Scala/Julia):
Python

MXNet version:
Or if installed from source:
0.10.1

MXNet commit hash (git rev-parse HEAD):
3ceb6d2

Python version and distribution:
Python 2.7.12
pip 9.0.1 from /usr/local/lib/python2.7/dist-packages (python 2.7)

What am I trying to achieve

I wanted to speed up predictions on a bunch of urls in order to scale. So I decided to preload the model.
I setup a server using rpyc package that preloads the model. I have client.py that connects to the server and predicts the result.
All of this is working fine but when I 'load test using apache bench' it gives me the following error.

Error Message:

Thread 2366 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffa8a09700 (LWP 22485)]
0x00007fffe4b55123 in mxnet::CopyFromTo(mxnet::NDArray const&, mxnet::NDArray*, int) ()
   from /home/ubuntu/softwares/mxnet/python/mxnet/../../lib/libmxnet.so
(gdb) 

I tried

  1. gdb to understand the error.
  2. htop to see how its performing

Finally, I would try loading the model normally instead of pre-loading.

Let me know what can be done!
Thanks,

@fasahath
Copy link
Author

fasahath commented Jun 29, 2017

Stumbled upon this #3946 issue and understood the problem much better.
Got to know that the engine is not thread-safe!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant