Linking of bin/vizdoom fails

[floringogianu]

As mentioned in #311 I was able to compile ViZDoom in a conda virtual env with gcc-5.5. I tried now doing the same on a server with ubuntu 16.04 and gcc-5.4 to find out that it fails to link the executable. edit: Apparently it fails due to some references to stricmp which are Windows specific. Any suggestions on how I can fix this?

The last few lines of the make output:

[ 99%] Linking CXX executable ../../../bin/vizdoom
CMakeFiles/vizdoom.dir/tempfiles.cpp.o: In function `FTempFileName::FTempFileName(char const*)':
tempfiles.cpp:(.text+0x7): warning: the use of `tempnam' is dangerous, better use `mkstemp'
/home/fgogianu/.conda/envs/wintermute/lib/libharfbuzz.so.0: undefined reference to `FT_Get_Var_Blend_Coordinates'
collect2: error: ld returned 1 exit status
src/vizdoom/src/CMakeFiles/vizdoom.dir/build.make:8920: recipe for target 'bin/vizdoom' failed
make[2]: *** [bin/vizdoom] Error 1
CMakeFiles/Makefile2:937: recipe for target 'src/vizdoom/src/CMakeFiles/vizdoom.dir/all' failed
make[1]: *** [src/vizdoom/src/CMakeFiles/vizdoom.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

The cmake output:

(wintermute) ➜  ViZDoom git:(master) ✗ cmake -DCMAKE_BUILD_TYPE=Release \                                   
-DBUILD_PYTHON3=ON -DBUILD_JAVA=OFF -DBUILD_LUA=OFF \
-DNUMPY_INCLUDES=/home/fgogianu/.conda/envs/wintermute/lib/python3.6/site-packages/numpy/core/include/ \
-DPYTHON_EXECUTABLE=/home/fgogianu/.conda/envs/wintermute/bin/python/ \
-DPYTHON_INCLUDE_DIR=/home/fgogianu/.conda/envs/wintermute/include/python3.6m/ \
-DPYTHON_LIBRARY=/home/fgogianu/.conda/envs/wintermute/lib/libpython3.6m.so

-- Boost version: 1.58.0
-- Found the following Boost libraries:
--   filesystem
--   thread
--   system
--   date_time
--   chrono
--   regex
--   iostreams
--   atomic
-- Could NOT find GME (missing:  GME_INCLUDE_DIR) 
-- Using system zlib
-- Using system jpeg library
-- Using system bzip2 library
-- Using internal gme library
-- /home/fgogianu/.conda/envs/wintermute/include
-- /opt/anaconda3/include
-- /home/fgogianu/tools/pip_packages/ViZDoom/src/vizdoom/lzma/C
-- Found SDL2: /usr/lib/x86_64-linux-gnu/libSDL2main.a;/usr/lib/x86_64-linux-gnu/libSDL2.so;-lpthread  
-- Found OpenAL: /usr/lib/x86_64-linux-gnu/libopenal.so  
-- Could not find FMOD include files
-- Could NOT find SndFile (missing:  SNDFILE_LIBRARY SNDFILE_INCLUDE_DIR) 
-- Could NOT find MPG123 (missing:  MPG123_LIBRARIES MPG123_INCLUDE_DIR) 
-- Found FluidSynth: /usr/lib/x86_64-linux-gnu/libfluidsynth.so  
-- Selected assembler: /usr/bin/as
-- Looking for filelength
-- Looking for filelength - not found
-- Looking for strupr
-- Looking for strupr - not found
-- Looking for stricmp
-- Looking for stricmp - not found
-- Looking for strnicmp
-- Looking for strnicmp - not found
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - found
-- Performing Test HAS_VA_COPY
-- Performing Test HAS_VA_COPY - Success
-- Boost version: 1.58.0
-- Found the following Boost libraries:
--   thread
--   system
--   date_time
--   chrono
--   atomic
-- Fluid synth libs: /usr/lib/x86_64-linux-gnu/libfluidsynth.so
-- Found PythonInterp: /home/fgogianu/.conda/envs/wintermute/bin/python (found suitable version "3.6.5", minimum required is "3") 
-- Found PythonLibs: /home/fgogianu/.conda/envs/wintermute/lib/libpython3.6m.so (found suitable version "3.6.5", minimum required is "3.6") 
-- Found NUMPY: /home/fgogianu/.conda/envs/wintermute/lib/python3.6/site-packages/numpy/core/include  
-- Found components for NumPy
-- NUMPY_ROOT_DIR    = /usr/local
-- NUMPY_INCLUDES    = /home/fgogianu/.conda/envs/wintermute/lib/python3.6/site-packages/numpy/core/include
-- NUMPY_LIBRARIES   = 
-- NUMPY_API_VERSION = 1.14.2
-- Found PythonInterp: /home/fgogianu/.conda/envs/wintermute/bin/python (found version "3.6.5") 
-- Found PythonLibs: /home/fgogianu/.conda/envs/wintermute/lib/libpython3.6m.so
-- Performing Test HAS_CPP14_FLAG
-- Performing Test HAS_CPP14_FLAG - Success
-- pybind11 v2.2.2
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- LTO enabled
-- Configuring done
-- Generating done
-- Build files have been written to: /home/fgogianu/tools/pip_packages/ViZDoom

And here is the full error log file CMakeFiles/CMakeError.log

[floringogianu]

pip install vizdoom also fails during linking with this:

    [ 99%] Linking CXX executable ../../../bin/vizdoom
    CMakeFiles/vizdoom.dir/tempfiles.cpp.o: In function `FTempFileName::FTempFileName(char const*)':
    tempfiles.cpp:(.text+0x7): warning: the use of `tempnam' is dangerous, better use `mkstemp'
    /opt/anaconda3/lib/libharfbuzz.so.0: undefined reference to `FT_Get_Var_Blend_Coordinates'
    collect2: error: ld returned 1 exit status
    src/vizdoom/src/CMakeFiles/vizdoom.dir/build.make:8894: recipe for target 'bin/vizdoom' failed
    make[2]: *** [bin/vizdoom] Error 1
    CMakeFiles/Makefile2:937: recipe for target 'src/vizdoom/src/CMakeFiles/vizdoom.dir/all' failed
    make[1]: *** [src/vizdoom/src/CMakeFiles/vizdoom.dir/all] Error 2
    make[1]: *** Waiting for unfinished jobs....
    [ 99%] Linking CXX shared module bin/python3.6/vizdoom.cpython-36m-x86_64-linux-gnu.so
    [ 99%] Built target libvizdoom_python
    Makefile:83: recipe for target 'all' failed
    make: *** [all] Error 2

[floringogianu]

I now think that what is causing the linking to fail is this: /home/fgogianu/.conda/envs/wintermute/lib/libharfbuzz.so.0: undefined reference to FT_Get_Var_Blend_Coordinates'
collect2: error: ld returned 1 exit status`. I updated the issue to reflect this.

Apparently the libharfbuzz and libfreetype are sometimes out of sync / incompatible.

On the machine where the build succeeds i have these versions:

> ldd bin/vizdoom
libharfbuzz.so.0 => /usr/lib/x86_64-linux-gnu/libharfbuzz.so.0 sym link to libharfbuzz.so.0.10702.0
libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 sym link to  libfreetype.so.6.15.0

On the machine where the linking fails I have these versions:

/home/fgogianu/.conda/envs/wintermute/lib/libharfbuzz.so.0: undefined reference to `FT_Get_Var_Blend_Coordinates'
/home/fgogianu/.conda/envs/wintermute/lib/libharfbuzz.so.0 sym link to libharfbuzz.so.0.10706.0
> ldd /home/fgogianu/.conda/envs/wintermute/lib/libharfbuzz.so.0
/home/fgogianu/.conda/envs/wintermute/lib/./libfreetype.so.6 sym link to libfreetype.so.6.14.0

[mwydmuch]

Hello @floringogianu,
I can only confirm that this is probably a bug that affects some range of versions of libharfbuzz. The only solution is to upgraded or downgraded the library.

[floringogianu]

Hi, thanks for the quick reply. Actually I believe this is a deeper issue. I was eventually able to compile it after created a clean miniconda installation for which I checked there are no libharfbuzz or libfreetype libraries in the root or in the envs. This way cmake linked to the (much older) versions in \usr\lib.

I did this after I downgraded the libharfbuzz in my anaconda env with no success. What I believe it is happening, but I couldn't figure out how to check it it is that while conda properly manages the libharfbuzz and libfreetype versions, ViZDoom somehow links to the system libfreetype and not the conda one.

[agiantwhale]

Hmm weird, same issue is happening for me. Works fine locally in OSX, but in Ubuntu 16.04 fails.

[MattiasMM]

@agiantwhale @floringogianu Did you ever find a solution to this problem?

[floringogianu]

@MattiasMM The "fix" for me was to create a new conda env with nothing in it besides the stuff required to build vizdoom. I was especially careful to make sure there were no libharfbuzz or libfreetype libraries in the conda root or in the envs (these are installed by opencv, for example). However this might not help if your linux distribution has different versions of these libraries.

[MattiasMM]

@floringogianu If I create a clean anaconda env (conda create --name py37 python=3.7) and manually remove all libharfbuzz.* files from my anaconda folder (what would the uninstall command be?), pip install vizdoom is said to complete successfully (Successfully installed vizdoom-1.1.6 it says in the command prompt). However, in runtime I cannot import vizdoom generates "ModuleNotFoundError". Any ideas what to check?

[MattiasMM]

Update: Python=3.6 executes import vizdoom fine (3.7 not). Same command under ipython generates ModuleNotFoundError. It will be good enough for my purposes, but it is a bit dissatisfying not to understand the reason for this behavior.

[MattiasMM]

One step on the way: If I give the anaconda path to Ipython sys.path.append('/home/mattiasm/anaconda3/envs/py36/lib/python3.6/site-packages/') I get the informative error: "This interpreter version: '3.6.5' doesn't match with version of the interpreter ViZDoom was compiled with: 3.6.7

[MattiasMM]

Finally! The mistake: jupyter and ipython were not installed with Conda and therefore their associated python distributions were not in sync with the conda python installation that was used to install vizdoom.

[MattiasM80]

@floringogianu It seems that without harfbuzz game.init() executes with an exception. Did you experience this and/or what version of harfbuzz did you use?

[floringogianu]

See here: #312 (comment) .

[phizaz]

I have a similar problem with this during pip install vizdoom. (with anaconda)

The error was:

/home/ta/anaconda3/lib/libharfbuzz.so.0: undefined reference to `FT_Done_MM_Var'
/home/ta/anaconda3/lib/libharfbuzz.so.0: undefined reference to `FT_Get_Var_Blend_Coordinates'

It is actually related to this thread openai/doom-py#16.

It is suggested that harfbuzz is the cause. My harfbuzz version before was 1.8.8 (latest in anaconda channel, but the latest is 2.3). After removed, I could successfully build vizdoom.

[flavienbwk]

I've installed libsndfile-dev and it solved the problem (Ubuntu).

[astronautas]

I've managed to fix this by installing vizdoom in miniconda instead of anaconda environment (clean Ubuntu 16.04, Python 3.6.7) and by doing pip install vizdoom. This way none of harfbuzz or freetype stuff get pulled into the anaconda folder and the system ones are used for the installation.

[mwydmuch]

The building system has been updated since then, manylinux wheels are now provided, closing the issue.