[Caffe2] Compile Error in ATen tests.

[xkszltl]

Issue description

I've been getting following errors on Linux CUDA build for quite a while.
Used to bypass them temporarily by BUILD_ATEN=OFF since I don't have time to dig in further.
But that flag has gone recently and I wanna know if there's any help I can get.

Here's the error log:

[108/114] cd /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/integer_divider_test.dir/__/aten/src/ATen/test && /usr/local/bin/cmake -E make_directory /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/integer_divider_test.dir/__/aten/src/ATen/test/. && /usr/local/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=Release -D generated_file:STRING=/media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/integer_divider_test.dir/__/aten/src/ATen/test/./integer_divider_test_generated_integer_divider_test.cu.o -D generated_cubin_file:STRING=/media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/integer_divider_test.dir/__/aten/src/ATen/test/./integer_divider_test_generated_integer_divider_test.cu.o.cubin.txt -P /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/integer_divider_test.dir/__/aten/src/ATen/test/integer_divider_test_generated_integer_divider_test.cu.o.Release.cmake
FAILED: caffe2/CMakeFiles/integer_divider_test.dir/__/aten/src/ATen/test/integer_divider_test_generated_integer_divider_test.cu.o 
cd /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/integer_divider_test.dir/__/aten/src/ATen/test && /usr/local/bin/cmake -E make_directory /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/integer_divider_test.dir/__/aten/src/ATen/test/. && /usr/local/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=Release -D generated_file:STRING=/media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/integer_divider_test.dir/__/aten/src/ATen/test/./integer_divider_test_generated_integer_divider_test.cu.o -D generated_cubin_file:STRING=/media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/integer_divider_test.dir/__/aten/src/ATen/test/./integer_divider_test_generated_integer_divider_test.cu.o.cubin.txt -P /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/integer_divider_test.dir/__/aten/src/ATen/test/integer_divider_test_generated_integer_divider_test.cu.o.Release.cmake
/opt/rh/devtoolset-7/root/usr/include/c++/7/bits/stl_pair.h(331): error: identifier "std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_empty_rep_storage" is undefined in device code

/opt/rh/devtoolset-7/root/usr/include/c++/7/bits/stl_pair.h(331): error: identifier "std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_empty_rep_storage" is undefined in device code

2 errors detected in the compilation of "/tmp/tmpxft_00017b4e_00000000-6_integer_divider_test.cpp1.ii".
CMake Error at integer_divider_test_generated_integer_divider_test.cu.o.Release.cmake:279 (message):
  Error generating file
  /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/integer_divider_test.dir/__/aten/src/ATen/test/./integer_divider_test_generated_integer_divider_test.cu.o


[109/114] cd /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_optional_test.dir/__/aten/src/ATen/test && /usr/local/bin/cmake -E make_directory /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_optional_test.dir/__/aten/src/ATen/test/. && /usr/local/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=Release -D generated_file:STRING=/media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_optional_test.dir/__/aten/src/ATen/test/./cuda_optional_test_generated_cuda_optional_test.cu.o -D generated_cubin_file:STRING=/media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_optional_test.dir/__/aten/src/ATen/test/./cuda_optional_test_generated_cuda_optional_test.cu.o.cubin.txt -P /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_optional_test.dir/__/aten/src/ATen/test/cuda_optional_test_generated_cuda_optional_test.cu.o.Release.cmake
FAILED: caffe2/CMakeFiles/cuda_optional_test.dir/__/aten/src/ATen/test/cuda_optional_test_generated_cuda_optional_test.cu.o 
cd /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_optional_test.dir/__/aten/src/ATen/test && /usr/local/bin/cmake -E make_directory /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_optional_test.dir/__/aten/src/ATen/test/. && /usr/local/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=Release -D generated_file:STRING=/media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_optional_test.dir/__/aten/src/ATen/test/./cuda_optional_test_generated_cuda_optional_test.cu.o -D generated_cubin_file:STRING=/media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_optional_test.dir/__/aten/src/ATen/test/./cuda_optional_test_generated_cuda_optional_test.cu.o.cubin.txt -P /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_optional_test.dir/__/aten/src/ATen/test/cuda_optional_test_generated_cuda_optional_test.cu.o.Release.cmake
/opt/rh/devtoolset-7/root/usr/include/c++/7/bits/stl_pair.h(331): error: identifier "std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_empty_rep_storage" is undefined in device code

/opt/rh/devtoolset-7/root/usr/include/c++/7/bits/stl_pair.h(331): error: identifier "std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_empty_rep_storage" is undefined in device code

2 errors detected in the compilation of "/tmp/tmpxft_00017b59_00000000-6_cuda_optional_test.cpp1.ii".
CMake Error at cuda_optional_test_generated_cuda_optional_test.cu.o.Release.cmake:279 (message):
  Error generating file
  /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_optional_test.dir/__/aten/src/ATen/test/./cuda_optional_test_generated_cuda_optional_test.cu.o


[110/114] cd /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_half_test.dir/__/aten/src/ATen/test && /usr/local/bin/cmake -E make_directory /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_half_test.dir/__/aten/src/ATen/test/. && /usr/local/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=Release -D generated_file:STRING=/media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_half_test.dir/__/aten/src/ATen/test/./cuda_half_test_generated_cuda_half_test.cu.o -D generated_cubin_file:STRING=/media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_half_test.dir/__/aten/src/ATen/test/./cuda_half_test_generated_cuda_half_test.cu.o.cubin.txt -P /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_half_test.dir/__/aten/src/ATen/test/cuda_half_test_generated_cuda_half_test.cu.o.Release.cmake
FAILED: caffe2/CMakeFiles/cuda_half_test.dir/__/aten/src/ATen/test/cuda_half_test_generated_cuda_half_test.cu.o 
cd /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_half_test.dir/__/aten/src/ATen/test && /usr/local/bin/cmake -E make_directory /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_half_test.dir/__/aten/src/ATen/test/. && /usr/local/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=Release -D generated_file:STRING=/media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_half_test.dir/__/aten/src/ATen/test/./cuda_half_test_generated_cuda_half_test.cu.o -D generated_cubin_file:STRING=/media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_half_test.dir/__/aten/src/ATen/test/./cuda_half_test_generated_cuda_half_test.cu.o.cubin.txt -P /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_half_test.dir/__/aten/src/ATen/test/cuda_half_test_generated_cuda_half_test.cu.o.Release.cmake
/opt/rh/devtoolset-7/root/usr/include/c++/7/bits/stl_pair.h(331): error: identifier "std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_empty_rep_storage" is undefined in device code

/opt/rh/devtoolset-7/root/usr/include/c++/7/bits/stl_pair.h(331): error: identifier "std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_empty_rep_storage" is undefined in device code

2 errors detected in the compilation of "/tmp/tmpxft_00017b66_00000000-6_cuda_half_test.cpp1.ii".
CMake Error at cuda_half_test_generated_cuda_half_test.cu.o.Release.cmake:279 (message):
  Error generating file
  /media/Scratch/tmp.yhKcQoslrF/pytorch/build/caffe2/CMakeFiles/cuda_half_test.dir/__/aten/src/ATen/test/./cuda_half_test_generated_cuda_half_test.cu.o

System Info

• PyTorch or Caffe2: Caffe2
• How you installed PyTorch (conda, pip, source): source
• Build command you used (if compiling from source): cmake+ninja
• OS: CentOS 7
• PyTorch version: master
• CUDA/cuDNN version: 9.2 / 7.1.4
• GPU models and configuration: Volta
• GCC version (if compiling from source): 7.3.1
• CMake version: 3.12.1

[xkszltl]

BTW same error on gcc-6

[xkszltl]

Looks like std::string is somehow referenced in the device code.

[xkszltl]

Temporarily bypassed with ATEN_NO_TEST.

[myleott]

I ran into a similar issue and also had warning about libtiff earlier in my build log. The solution for me was to conda uninstall libtiff and update my LD_LIBRARY_PATH so that it does NOT include the anaconda lib/ directory (based on the suggestion here: BVLC/caffe#1559 (comment)).

[soumith]

a patch should land for this tomorrow. cc: @orionr to close this issue once it lands.

[orionr]

This fix has landed on master - please verify it works for you.

[xkszltl]

@orionr
Could you tell me which commit fixes the issue?
I tried the latest master and all the errors are still there.

[orionr]

HI @xkszltl - apologies, I didn't let you know how to disable tests. You can now do

NO_TEST=1 python setup.py <build_deps|develop|install|etc>

Also, @myleott OpenCV should be disabled by default now. You can enable it with

USE_OPENCV=1 python setup.py <build_deps|develop|install|etc>

[xkszltl]

@orionr
Not really understand what you're talking about.

I'm building with CMake directly, and currently use -DBUILD_ATEN=OFF to bypass the issue.
I thought you're closing this because there's a bugfix from soumith to resolve the compile error, but when I tried the latest master all the errors are still there.