Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

MXNet built with GPerftools crashes #9096

Open
rahul003 opened this issue Dec 15, 2017 · 0 comments
Open

MXNet built with GPerftools crashes #9096

rahul003 opened this issue Dec 15, 2017 · 0 comments

Comments

@rahul003
Copy link
Member

Description

MXNet built using cmake with the flag USE_GPERFTOOLS=ON crashes on running any job with the error

Environment info (Required)

Note that I actually used the compiler gcc and g++ versions 5.4 with ccache although this script says 7.2

----------Python Info----------
Version      : 3.6.3
Compiler     : GCC 7.2.0
Build        : ('default', 'Oct 13 2017 12:02:49')
Arch         : ('64bit', '')
------------Pip Info-----------
Version      : 9.0.1
Directory    : /home/ubuntu/anaconda3/lib/python3.6/site-packages/pip
----------MXNet Info-----------
No MXNet installed.
----------System Info----------
Platform     : Linux-4.4.0-1043-aws-x86_64-with-debian-stretch-sid
system       : Linux
node         : ip-172-31-18-227
release      : 4.4.0-1043-aws
version      : #52-Ubuntu SMP Tue Dec 5 10:49:06 UTC 2017
----------Hardware Info----------
machine      : x86_64
processor    : x86_64
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                32
On-line CPU(s) list:   0-31
Thread(s) per core:    2
Core(s) per socket:    16
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Model name:            Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
Stepping:              1
CPU MHz:               1333.191
CPU max MHz:           3000.0000
CPU min MHz:           1200.0000
BogoMIPS:              4600.04
Hypervisor vendor:     Xen
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              46080K
NUMA node0 CPU(s):     0-31
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx xsaveopt
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0042 sec, LOAD: 0.4159 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.1394 sec, LOAD: 0.1323 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.1641 sec, LOAD: 0.4018 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0170 sec, LOAD: 0.1574 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0031 sec, LOAD: 0.1818 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0029 sec, LOAD: 0.0631 sec.

Package used : Python

Build info (Required if built from source)

cmake -DCMAKE_BUILD_TYPE=Release -DMKL_INCLUDE_DIR=/opt/intel/mkl/include -DMKLML_GNU_LIBRARY=../mkl/lib/libmklml_gnu.so -DUSE_OLDCMAKECUDA=ON -GNinja .. && ninja

Compiler (gcc/clang/mingw/visual studio): gcc

MXNet commit hash: 5540ee9

Build config:
(Paste the content of config.mk, or the build command.)

Error Message:

src/tcmalloc.cc:278] Attempt to free invalid pointer 0x55c5bcd9f780
Aborted (core dumped)

Minimum reproducible example

cd example/image-classification && PYTHONPATH=../../python python train_mnist.py

Steps to reproduce

(Paste the commands you ran that produced the error.)

  1. Build with cmake using GPERFTOOLS
  2. Run the above example
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants