ModuleNotFoundError: No module named 'vizdoom' on conda/venv env

[floringogianu]

Both pip install vizdoom and pip install . from sources terminate without error, however the module is somehow not registered:

$ python -c 'from vizdoom import *'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'vizdoom'

$ ls ~/miniconda3/envs/torch30/lib/python3.6/site-packages/vizdoom-1.1.5.dist-info
INSTALLER  METADATA  RECORD  top_level.txt  WHEEL

Any suggestions?

[floringogianu]

I was able to compile it and install it manually like this:

cmake -DCMAKE_BUILD_TYPE=Release \
-DBUILD_PYTHON3=ON -DBUILD_JAVA=OFF -DBUILD_LUA=OFF \
-DNUMPY_INCLUDES=~/Tools/miniconda3/envs/torch30/lib/python3.6/site-packages/numpy/core/include/ \
-DPYTHON_EXECUTABLE=~/Tools/miniconda3/envs/torch30/bin/python/ \
-DPYTHON_INCLUDE_DIR=~/Tools/miniconda3/envs/torch30/include/python3.6m/ \
-DPYTHON_LIBRARY=~/Tools/miniconda3/envs/torch30/lib/libpython3.6m.so

make

cp -r bin/python3/pip_package ~/Tools/miniconda3/envs/torch30/lib/python3.6/site-packages/vizdoom

I would leave the issue open though, I'd try to figure out why nor the PyPI method or the setup from sources is working.

[mihahauke]

Are you sure that the pip you used is the one for python you use?

[floringogianu]

I never considered checking because that's what conda virtualenv manages for you. But yes, it seems to be fine:

$ which python
/home/florin/Tools/miniconda3/envs/torch30/bin/python
$ which pip
/home/florin/Tools/miniconda3/envs/torch30/bin/pip

I asked a friend with a different ubuntu version to conda create -n test_env python==3.6; source activate test_env; pip install vizdoom and he gets the same error.

[biggzlar]

Problem persists. I set up a fresh 'virtualenv', installed all dependencies via pip3 and pip freezedoes show vizdoom==1.1.4 but importing fails:

(...)
urllib3==1.22
vizdoom==1.1.4
wcwidth==0.1.7
Werkzeug==0.14.1
wrapt==1.10.11
zmq==0.0.0

~/ViZDoom/
(env_3_6) [master…]$: python examples/python/fps.py
Traceback (most recent call last):
  File "fps.py", line 16, in <module>
    from vizdoom import *
ModuleNotFoundError: No module named 'vizdoom'

[mihahauke]

@biggzlar Could you give us the exact commands to create the virtualenv?

[biggzlar]

Thanks for your reply. It’s really just this:

$: virtualenv env_3_6 —python=python3.6
$: . env_3_6/bin/activate
$: pip install numpy vizdoom

No errors or anything.

[Miffyli]

Fascinating, I am getting the following when I first tried to replicate the error. However when I "uninstall" vizdoom and try to install it again (in the environment), it throws no errors and claims vizdoom is installed correctly.

@mihahauke
Maybe this is related to how sometimes people require to sudo the pip installation?

virtualenv testenv —python=python3.5
. testenv/bin/activate
pip install numpy vizdoom

  [ 99%] Building CXX object src/vizdoom/src/CMakeFiles/vizdoom.dir/viz_message_queue.cpp.o
  [ 99%] Building CXX object src/vizdoom/src/CMakeFiles/vizdoom.dir/viz_screen.cpp.o
  [ 99%] Building CXX object src/vizdoom/src/CMakeFiles/vizdoom.dir/viz_shared_memory.cpp.o
  /tmp/pip-install-r20q55g4/vizdoom/src/vizdoom/src/viz_main.cpp: In function ‘void VIZ_PrintFuncMsg(const char*, const char*)’:
  /tmp/pip-install-r20q55g4/vizdoom/src/vizdoom/src/viz_main.cpp:463:23: warning: NULL used in arithmetic [-Wpointer-arith]
       while (func[s] != NULL && func[s] != ' ') ++s;
                         ^
  /tmp/pip-install-r20q55g4/vizdoom/src/vizdoom/src/viz_main.cpp:465:23: warning: NULL used in arithmetic [-Wpointer-arith]
       while (func[e] != NULL && func[e] != '(') ++e;
                         ^
  [ 99%] Linking CXX executable ../../../bin/vizdoom
  CMakeFiles/vizdoom.dir/tempfiles.cpp.o: In function `FTempFileName::FTempFileName(char const*)':
  tempfiles.cpp:(.text+0x7): warning: the use of `tempnam' is dangerous, better use `mkstemp'
  [ 99%] Built target vizdoom
  Makefile:83: recipe for target 'all' failed
  make: *** [all] Error 2
  
  Installation failed, you may be missing some dependencies.
  Please check https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md for details
  
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-r20q55g4/vizdoom/setup.py", line 119, in <module>
      keywords=['vizdoom', 'doom', 'ai', 'deep learning', 'reinforcement learning', 'research']
    File "/home/anssk/testenv/testenv36/lib/python3.5/site-packages/setuptools/__init__.py", line 129, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/home/anssk/testenv/testenv36/lib/python3.5/site-packages/wheel/bdist_wheel.py", line 202, in run
      self.run_command('build')
    File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/tmp/pip-install-r20q55g4/vizdoom/setup.py", line 82, in run
      subprocess.check_call(['make', '-j', str(cpu_cores)])
    File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['make', '-j', '7']' returned non-zero exit status 2
  
  ----------------------------------------
  Failed building wheel for vizdoom
  Running setup.py clean for vizdoom
Failed to build vizdoom
Installing collected packages: numpy, vizdoom
  Running setup.py install for vizdoom ... done
Successfully installed numpy-1.14.5 vizdoom-1.1.4

[biggzlar]

That is weird. Also just tried to run the pip installation with sudo, again with the same result.

[Miffyli]

sudoing while in environment seems to use the global pip to install packages (or that's just my setup doing that). I tried to replicate the issue again but now everything seems to install fine, even with pip's --no-cache-dir option õ.o

What happens if you try to install vizdoom without the environment? Maybe there are some dependencies missing and something suppresses the errors somewhere?

[biggzlar]

Just tried, the import still fails even when installing without a virtual environment. :/

$: python3 -m pip install vizdoom
Collecting vizdoom
Requirement already satisfied: numpy in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from vizdoom)
Installing collected packages: vizdoom
Successfully installed vizdoom-1.1.4

$: python3 -m pip freeze
(...)
numpy==1.14.5
(...)
vizdoom==1.1.4
Werkzeug==0.14.1
wrapt==1.10.11

$: python3 -c 'import vizdoom'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'vizdoom'

[Miffyli]

Hmm that Python path looks peculiar, could be throwing off ViZDoom.

What do the following commands show for you (with and without virtualenv). I think the most perplexing part here is the fact pip claims ViZDoom is somewhere while Python itself can't find it:

python3 -m site
python3 -m site --user-site
python -c "import numpy; print(numpy.__file__)

[biggzlar]

Here are the outputs:

Without an active environment

$: python3 -m site
sys.path = [
    '/Users/biggzlar',
    '/Users/biggzlar/Library/Python/2.7/bin',
    '/Library/Frameworks/Python.framework/Versions/3.6/lib/python36.zip',
    '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6',
    '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload',
    '/Users/biggzlar/Library/Python/3.6/lib/python/site-packages',
    '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages',
]
USER_BASE: '/Users/biggzlar/Library/Python/3.6' (exists)
USER_SITE: '/Users/biggzlar/Library/Python/3.6/lib/python/site-packages' (exists)
ENABLE_USER_SITE: True

$: python3 -m site --user-site
/Users/biggzlar/Library/Python/3.6/lib/python/site-packages

$: python -c "import numpy; print(numpy.__file__)"
/usr/local/lib/python2.7/site-packages/numpy/__init__.pyc

$: python3 -c "import numpy; print(numpy.__file__)"
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/__init__.py

I see what you mean when you say peculiar...
Now the same in an environment:

Active Python 3.6 environment

$: python3 -m site
sys.path = [
    '/Users/biggzlar',
    '/Users/biggzlar/Library/Python/2.7/bin',
    '/Users/biggzlar/env3/lib/python36.zip',
    '/Users/biggzlar/env3/lib/python3.6',
    '/Users/biggzlar/env3/lib/python3.6/lib-dynload',
    '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6',
    '/Users/biggzlar/env3/lib/python3.6/site-packages',
]
USER_BASE: '/Users/biggzlar/.local' (exists)
USER_SITE: '/Users/biggzlar/.local/lib/python3.6/site-packages' (exists)
ENABLE_USER_SITE: False

$: python3 -m site --user-site
/Users/biggzlar/.local/lib/python3.6/site-packages

$: python -c "import numpy; print(numpy.__file__)"
/Users/biggzlar/env3/lib/python3.6/site-packages/numpy/__init__.py

[Miffyli]

Check if you have any vizdoom directories in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages. Seems like pip only cares if the vizdoom-1.1.4.dist-info directory is present, even if the library itself (vizdoom) is missing. At this point it sounds like compiling goes well but for some reason the compiled library is not added to Python path. You could try manual compiling like above, should be straight-forward enough.

[biggzlar]

After uninstalling with pip there are no vizdoom related folders left in both environments. As for the compilation, I am actually now encountering this issue #236

Thank you for your help so far!

[Miffyli]

I have no experience with Mac stuff, but could this be the issue?

https://stackoverflow.com/questions/16454129/python-h-header-file-missing-on-mac-os-x-10-6
(replace #include <Python.h> with #include <Python/Python.h> and add -framework Python to gcc parameters).

[biggzlar]

It very well could be, but the solutions don't seem to work. Added the relevant parameter flags to the CMakeLists.txt and ran the build process again as described above - still missing the file.

I then updated the include paths in my current environment, pointing to the headers folder as such:

export C_INCLUDE_PATH=/System/Library/Frameworks/Python.framework/Headers
export CPLUS_INCLUDE_PATH=/System/Library/Frameworks/Python.framework/Headers

No luck.

[Miffyli]

Things go beyond me at this point :(. From what I gathered around internet the problem seems to the wrong include statement. I would love to debug this myself but closest Mac computer is likely few dozen kilometers away from me currently.

[biggzlar]

Thanks for your help either way! I'll need to switch machines for now but keep looking into the issue.

[domschl]

The problem persists (Arch Linux, Python 3.7).
It seems to be caused by Python version naming-confusions when building the PIP package: part of the make procedure copies things to bin/python3 and other parts reference bin/python3.7. This causes silent errors and generates a broken PIP package that nevertheless installs without error.

[100%] Built target pk3
Scanning dependencies of target python_pip_package
[100%] Assembling pip package in ./ViZDoom/bin/python3/pip_package
[100%] Built target python_pip_package
running build_py
package init file 'bin/python3.7/pip_package/__init__.py' not found (or not a regular file)
running egg_info

I've made a quick'n dirty fix by creating a link ln -s python3.7 python3 in ViZDoom/bin, and then the pip package is actually build successful (and has actual content).

Once the package was installed, I encountered #333 (Cython version-check breaks on 3.7!=3.7.0). After patching __init__.py as suggested in the report and rebuilding it worked fine.