Error with python-rocksdb while trying to build

[Maxino22]

Hello trying to setup indy-node first time I followed steps cloned and tried to use .DevContainer however I'm getting errors with at the end python-rocks db setup is not building and crushes everything. even running pip install python-rocksdb

rocksdb/_rocksdb.pyx:2413:37: Constructing Python tuple not allowed without gil
  
  Error compiling Cython file:
  ------------------------------------------------------------
  ...
  
      def get_backup_info(self):
          cdef vector[backup.BackupInfo] backup_info
  
          with nogil:
              self.engine.GetBackupInfo(cython.address(backup_info))
                                              ^
  ------------------------------------------------------------
  
  rocksdb/_rocksdb.pyx:2413:44: Converting to Python object not allowed without gil
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-wzq_8yme/python-rocksdb/setup.py", line 21, in <module>
      setup(
    File "/usr/local/lib/python3.8/dist-packages/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.8/dist-packages/setuptools/command/install.py", line 61, in run
      return orig.install.run(self)
    File "/usr/lib/python3.8/distutils/command/install.py", line 589, in run
      self.run_command('build')
    File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/lib/python3.8/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.8/dist-packages/setuptools/command/build_ext.py", line 79, in run
      _build_ext.run(self)
    File "/usr/lib/python3.8/distutils/command/build_ext.py", line 340, in run
      self.build_extensions()
    File "/usr/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
      self._build_extensions_serial()
    File "/usr/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
      self.build_extension(ext)
    File "/usr/local/lib/python3.8/dist-packages/setuptools/command/build_ext.py", line 196, in build_extension
      _build_ext.build_extension(self, ext)
    File "/home/vscode/.local/lib/python3.8/site-packages/Cython/Distutils/build_ext.py", line 130, in build_extension
      new_ext = cythonize(
    File "/home/vscode/.local/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 1154, in cythonize
      cythonize_one(*args)
    File "/home/vscode/.local/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 1321, in cythonize_one
      raise CompileError(None, pyx_file)
  Cython.Compiler.Errors.CompileError: rocksdb/_rocksdb.pyx
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-wzq_8yme/python-rocksdb/setup.py'"'"'; __file__='"'"'/tmp/pip-install-wzq_8yme/python-rocksdb/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-asiu2aol/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/vscode/.local/include/python3.8/python-rocksdb Check the logs for full command output.

Kindly assist

[smqik]

Hi, even I am getting this error when building indy-node in the dev-container. I tried building the node both in Windows (using WSL), and Ubuntu 16.04, 20.04 LTS, and I am getting the same error. The python-rocksdb package is throwing this error. I even tried building rocksdb from the source, but still faced the same issue.

[WadeBarnes]

@Maxino22, @smqik, I've tried this on Windows, and it's working for me. I selected the "Rebuild Container Without Cache" option and the devcontainer builds and launches fine. I did need to update the devcontainer.json file to the latest format. PR incoming. See #1824.

Would either of you be able to provide me with a complete build log?

[WadeBarnes]

Ok, I see. The issue is with the post install when it's installing and processing the pip install .[tests] command.

[Maxino22]

hello yes that was the issue and is related with python-rocksdb ?

[smqik]

I tried building the node again after updating devcontainer.json. Still getting the same error.

[WadeBarnes]

I think I found the issue. It seems to be an update to Cython causing the issue. If I pip install Cython==0.29.36 I can then pip install python-rocksdb.

I'm testing out this change to the .devcontainer\Dockerfile:

RUN pip3 install -U \
    # Required by setup.py
    setuptools==50.3.2 \
    'pyzmq==22.3.0'

to

RUN pip3 install -U \
    # Required by setup.py
    setuptools==50.3.2 \
    'pyzmq==22.3.0' \
    Cython==0.29.36

[WadeBarnes]

Ok, That is working for me. I'll add that to the PR. This is just a workaround as the Cython - python-rocksdb python package dependency is actually at the indy-plenum level. I'll enter a ticket for that.

[WadeBarnes]

The PR has been updated with the workaround.

[Maxino22]

nice I tried with the Cython version provided it works