core dump in running tensorflow benchmark

[myotheone]

I have successfully installed byteps using "python setup.py install".
when i run benchmark, byteps core dumped.

core backtrack:

env:
1.tf version 1.14
2.cuda version: 9.0
3.nccl version: 2.4.7 for cuda9.0
4.os: ubuntu 16.04
5.g++: 5.4.0

script: copy from step_by_step_tutorial.md
export LD_LIBRARY_PATH=/usr/local/cuda-9.0/lib64/:$LD_LIBRARY_PATH
export NVIDIA_VISIBLE_DEVICES=0,1,2,3
export DMLC_WORKER_ID=0
export DMLC_NUM_WORKER=1
export DMLC_ROLE=worker
export DMLC_NUM_SERVER=1
export DMLC_PS_ROOT_URI=x.x.x.x
export DMLC_PS_ROOT_PORT=9999
export EVAL_TYPE=benchmark

python /home/mark/mark/code/byteps/launcher/launch.py \
/home/mark/mark/code/byteps/example/tensorflow/run_tensorflow_byteps.sh \
--model ResNet50 --num-iters 1000

[ymjiang]

Looks like your gcc version is higher than our tested gcc-4.9.

Can you try to pin down gcc before you build BytePS? Here is an example: https://github.com/bytedance/byteps/blob/master/docker/Dockerfile.worker.tensorflow#L115-L123

[bobzhuyb]

I think it's a very similar issue as this tensorflow/tensorflow#13308 (comment)
It happens when BytePS is compiled with gcc 5 while TF is compiled with gcc 4.

It's still an open issue..

If your TF is compiled with gcc 4 (and it seems so), you have to use gcc-4.9 to build BytePS. You can try the suggestion given by @ymjiang , or use our pre-built docker image.

[ymjiang]

@un-knight Can you share more information about your OS and env?

[bobzhuyb]

@un-knight Would you reply to the issue thread you already opened #20? From your log, I don't see how your question is related to this issue.

[byronyi]

Just provide the binary release. No need to build one on users' environment, we do not need mpicc or mpicxx.

[bobzhuyb]

Our plan is to release binary installation package built for different frameworks/CUDA versions, in order to ease user installation process.

[ymjiang]

@myotheone We just uploaded some pypi lists for easier installation. See https://github.com/bytedance/byteps/blob/master/docs/pip-list.md

[bobzhuyb]

Closing this since we started providing pypi packages. Feel free to reopen.