Path to CuDNN library not exists

[s3p02]

I have a 1060 and a K40 installed. I use Nvidia Digits, and have CuDNN installed.

I changed the setup file to:

CUDNN_PATH=${HOME}/usr/local/cuda

I get the error :

2 GPU CUDA device(s) found

-- GPU device exists
CMake Error at CMakeLists.txt:63 (message):
Path to CuDNN library not exists

-- Configuring incomplete, errors occurred!
See also "/usr/local/DNNMark/build/CMakeFiles/CMakeOutput.log".

How do I resolve this?

/usr/local$ ls | grep cu

Gives me:

cub-1.4.1
cuda
cuda-8.0
cudnn-5.1
cudnn-8.0-linux-x64-v5.1.tgz

[shidong-ai]

Hi,

You should replace the entire "${HOME}/usr/local/cuda" with the absolute path where your cudnn is stored.
In your case, you can try to change that line into CUDNN_PATH=/usr/local/cudnn-5.1
Let me know if you have other problems.

[s3p02]

Should I be worried about Warnings?

2 GPU CUDA device(s) found

-- GPU device exists
-- CuDNN library root path: /usr/local/cudnn-5.1
-- DNNMark Source files: core/src/dnn_utility.cccore/src/utility.cccore/src/gpu_utility.cccore/src/dnn_config_keywords.cccore/src/common.cccore/src/dnnmark.cc
-- dnnmark_test_fwd_conv: Benchmark source files: test_fwd_conv.cc
-- dnnmark_test_bwd_conv: Benchmark source files: test_bwd_conv.cc
-- dnnmark_test_fwd_pool: Benchmark source files: test_fwd_pool.cc
-- dnnmark_test_bwd_pool: Benchmark source files: test_bwd_pool.cc
-- dnnmark_test_fwd_lrn: Benchmark source files: test_fwd_lrn.cc
-- dnnmark_test_bwd_lrn: Benchmark source files: test_bwd_lrn.cc
-- dnnmark_test_fwd_activation: Benchmark source files: test_fwd_activation.cc
-- dnnmark_test_bwd_activation: Benchmark source files: test_bwd_activation.cc
-- dnnmark_test_fwd_fc: Benchmark source files: test_fwd_fc.cc
-- dnnmark_test_bwd_fc: Benchmark source files: test_bwd_fc.cc
-- dnnmark_test_fwd_softmax: Benchmark source files: test_fwd_softmax.cc
-- dnnmark_test_bwd_softmax: Benchmark source files: test_bwd_softmax.cc
-- dnnmark_test_composed_model: Benchmark source files: test_composed_model.cc
-- Configuring done
CMake Warning at /usr/share/cmake-2.8/Modules/FindCUDA.cmake:1441 (add_library):
Cannot generate a safe runtime search path for target dnnmark because files
in some directories may conflict with libraries in implicit directories:

runtime library [libcudnn.so.5] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /usr/local/cuda/lib64

Some of these libraries may not be found correctly.
Call Stack (most recent call first):
CMakeLists.txt:95 (cuda_add_library)

CMake Warning at benchmarks/test_fwd_conv/CMakeLists.txt:22 (add_executable):
Cannot generate a safe runtime search path for target dnnmark_test_fwd_conv
because files in some directories may conflict with libraries in implicit
directories:

runtime library [libcudnn.so.5] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /usr/local/cuda/lib64

Some of these libraries may not be found correctly.

CMake Warning at benchmarks/test_bwd_conv/CMakeLists.txt:22 (add_executable):
Cannot generate a safe runtime search path for target dnnmark_test_bwd_conv
because files in some directories may conflict with libraries in implicit
directories:

runtime library [libcudnn.so.5] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /usr/local/cuda/lib64

Some of these libraries may not be found correctly.

CMake Warning at benchmarks/test_fwd_pool/CMakeLists.txt:19 (add_executable):
Cannot generate a safe runtime search path for target dnnmark_test_fwd_pool
because files in some directories may conflict with libraries in implicit
directories:

runtime library [libcudnn.so.5] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /usr/local/cuda/lib64

Some of these libraries may not be found correctly.

CMake Warning at benchmarks/test_bwd_pool/CMakeLists.txt:19 (add_executable):
Cannot generate a safe runtime search path for target dnnmark_test_bwd_pool
because files in some directories may conflict with libraries in implicit
directories:

runtime library [libcudnn.so.5] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /usr/local/cuda/lib64

Some of these libraries may not be found correctly.

CMake Warning at benchmarks/test_fwd_lrn/CMakeLists.txt:19 (add_executable):
Cannot generate a safe runtime search path for target dnnmark_test_fwd_lrn
because files in some directories may conflict with libraries in implicit
directories:

runtime library [libcudnn.so.5] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /usr/local/cuda/lib64

Some of these libraries may not be found correctly.

CMake Warning at benchmarks/test_bwd_lrn/CMakeLists.txt:19 (add_executable):
Cannot generate a safe runtime search path for target dnnmark_test_bwd_lrn
because files in some directories may conflict with libraries in implicit
directories:

runtime library [libcudnn.so.5] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /usr/local/cuda/lib64

Some of these libraries may not be found correctly.

CMake Warning at benchmarks/test_fwd_activation/CMakeLists.txt:19 (add_executable):
Cannot generate a safe runtime search path for target
dnnmark_test_fwd_activation because files in some directories may conflict
with libraries in implicit directories:

runtime library [libcudnn.so.5] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /usr/local/cuda/lib64

Some of these libraries may not be found correctly.

CMake Warning at benchmarks/test_bwd_activation/CMakeLists.txt:19 (add_executable):
Cannot generate a safe runtime search path for target
dnnmark_test_bwd_activation because files in some directories may conflict
with libraries in implicit directories:

runtime library [libcudnn.so.5] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /usr/local/cuda/lib64

Some of these libraries may not be found correctly.

CMake Warning at benchmarks/test_fwd_fc/CMakeLists.txt:19 (add_executable):
Cannot generate a safe runtime search path for target dnnmark_test_fwd_fc
because files in some directories may conflict with libraries in implicit
directories:

runtime library [libcudnn.so.5] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /usr/local/cuda/lib64

Some of these libraries may not be found correctly.

CMake Warning at benchmarks/test_bwd_fc/CMakeLists.txt:19 (add_executable):
Cannot generate a safe runtime search path for target dnnmark_test_bwd_fc
because files in some directories may conflict with libraries in implicit
directories:

runtime library [libcudnn.so.5] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /usr/local/cuda/lib64

Some of these libraries may not be found correctly.

CMake Warning at benchmarks/test_fwd_softmax/CMakeLists.txt:19 (add_executable):
Cannot generate a safe runtime search path for target
dnnmark_test_fwd_softmax because files in some directories may conflict
with libraries in implicit directories:

runtime library [libcudnn.so.5] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /usr/local/cuda/lib64

Some of these libraries may not be found correctly.

CMake Warning at benchmarks/test_bwd_softmax/CMakeLists.txt:19 (add_executable):
Cannot generate a safe runtime search path for target
dnnmark_test_bwd_softmax because files in some directories may conflict
with libraries in implicit directories:

runtime library [libcudnn.so.5] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /usr/local/cuda/lib64

Some of these libraries may not be found correctly.

CMake Warning at benchmarks/test_composed_model/CMakeLists.txt:22 (add_executable):
Cannot generate a safe runtime search path for target
dnnmark_test_composed_model because files in some directories may conflict
with libraries in implicit directories:

runtime library [libcudnn.so.5] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /usr/local/cuda/lib64

Some of these libraries may not be found correctly.

-- Generating done
-- Build files have been written to: /usr/local/DNNMark/build

It would be great if y'all could publish a detailed tutorial. Thanks

[shidong-ai]

Thank you for your advice! Publishing a tutorial is a good idea.
Back to the warning. It seems that the warning suggests that you have two copy of cuda related library. Could you run 'ls /usr/lib/x86_64-linux-gnu | grep cu' to see if there is any shared library installed there? And meanwhile, can you try to run make to see if there is any complaint?
Thanks!

[s3p02]

'ls /usr/lib/x86_64-linux-gnu | grep cucups-pk-helper-mechanism' gives:

icu
libcucul.so.0
libcucul++.so.0
libcucul.so.0.99.18
libcucul++.so.0.99.18
libcuda.so
libcuda.so.1
libcuda.so.340.101
libcuda.so.375.39
libcudnn.so
libcudnn.so.5
libcudnn.so.5.1.10
libcudnn_static.a
libcudnn_static_v5.a
libcupscgi.so.1
libcupsfilters.so.1
libcupsfilters.so.1.0.0
libcupsimage.so.2
libcupsmime.so.1
libcupsppdc.so.1
libcups.so.2
libcurl.a
libcurl-gnutls.a
libcurl-gnutls.la
libcurl-gnutls.so
libcurl-gnutls.so.3
libcurl-gnutls.so.4
libcurl-gnutls.so.4.3.0
libcurl.la
libcurl.so
libcurl.so.3
libcurl.so.4
libcurl.so.4.3.0
libcurses.a
libcurses.so
libharfbuzz-icu.so
libharfbuzz-icu.so.0
libharfbuzz-icu.so.0.927.0
libicudata.a
libicudata.so
libicudata.so.52
libicudata.so.52.1
libicui18n.a
libicui18n.so
libicui18n.so.52
libicui18n.so.52.1
libicuio.a
libicuio.so
libicuio.so.52
libicuio.so.52.1
libicule.a
libicule.so
libicule.so.52
libicule.so.52.1
libiculx.a
libiculx.so
libiculx.so.52
libiculx.so.52.1
libicutest.a
libicutest.so
libicutest.so.52
libicutest.so.52.1
libicutu.a
libicutu.so
libicutu.so.52
libicutu.so.52.1
libicuuc.a
libicuuc.so
libicuuc.so.52
libicuuc.so.52.1
libncurses.a
libncurses++.a
libncurses.so
libncurses++w.a
libncursesw.a
libncursesw.so
libQt5Concurrent.prl
libQt5Concurrent.so
libQt5Concurrent.so.5
libQt5Concurrent.so.5.2
libQt5Concurrent.so.5.2.1
libvtkCommonExecutionModel-6.0.so
libvtkCommonExecutionModel-6.0.so.6.0
libvtkCommonExecutionModel-6.0.so.6.0.0
libvtkCommonExecutionModelJava.so
libvtkCommonExecutionModelPython27D-6.0.so
libvtkCommonExecutionModelPython27D-6.0.so.6.0
libvtkCommonExecutionModelPython27D-6.0.so.6.0.0
libvtkCommonExecutionModelTCL-6.0.so
libvtkCommonExecutionModelTCL-6.0.so.6.0
libvtkCommonExecutionModelTCL-6.0.so.6.0.0
libwayland-cursor.a
libwayland-cursor.so
libwayland-cursor.so.0
libwayland-cursor.so.0.0.0
libXcursor.a
libXcursor.so
libXcursor.so.1
libXcursor.so.1.0.2

[shidong-ai]

Just like I guessed. In this folder, I saw there are some duplicated cuda related libraries including cudnn installed here. I don't know how you did it, but I would suggest you clean out the cuda related library in /usr/lib/x86_64-linux-gnu and only use the one in /usr/local. I believe this can help you get rid of these warning messages. Nevertheless, if the version of these duplicated cuda libraries is the same, I don't see a problem in the building process.

[shidong-ai]

Close this issue and please submit another if you encountered another one. Thanks!

[hichamhendy]

Just like I guessed. In this folder, I saw there are some duplicated cuda related libraries including cudnn installed here. I don't know how you did it, but I would suggest you clean out the cuda related library in /usr/lib/x86_64-linux-gnu and only use the one in /usr/local. I believe this can help you get rid of these warning messages. Nevertheless, if the version of these duplicated cuda libraries is the same, I don't see a problem in the building process.

I have the same problem and I get the error 'CUDA Error: an illegal memory access was encountered' could it be related?