Unable to install benji to Python 3.11 venv in Proxmox 8 (Debian 12 (bookworm))

[bbs2web]

Hi,

The following thread references Python 3.11 being supported, am I missing something on a Debian 12 based system or should a new install work?
#153

apt-get install --no-install-recommends software-properties-common;
apt-get --no-install-recommends install gcc git libdpkg-perl python3-dev python3-pip python3-rados python3-rbd python3-venv;
apt-get install python3-markupsafe;
python3 -m venv /usr/local/benji;
ln -s /usr/lib/python3/dist-packages/rados.cpython-311-x86_64-linux-gnu.so /usr/local/benji/lib/python3.11/site-packages/rados.cpython-311-x86_64-linux-gnu.so;
ln -s /usr/lib/python3/dist-packages/rbd.cpython-311-x86_64-linux-gnu.so /usr/local/benji/lib/python3.11/site-packages/rbd.cpython-311-x86_64-linux-gnu.so;
. /usr/local/benji/bin/activate;
pip install benji[compression,s3,readcache,helpers];

Error:

(benji) [root@kvm1 ~]# pip install benji[compression,s3,readcache,helpers];
Collecting benji[compression,helpers,readcache,s3]
  Using cached benji-0.17.0-py3-none-any.whl (144 kB)
WARNING: benji 0.17.0 does not provide the extra 'readcache'
Collecting PrettyTable<1,>=0.7.2
  Using cached prettytable-0.7.2.zip (28 kB)
  Preparing metadata (setup.py) ... done
Collecting sqlalchemy<2,>=1.4.14
  Using cached SQLAlchemy-1.4.49-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
Collecting setproctitle<2,>=1.1.8
  Using cached setproctitle-1.3.2-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (31 kB)
Collecting python-dateutil<3,>=2.6.0
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting alembic<2,>=1.0.5
  Using cached alembic-1.12.0-py3-none-any.whl (226 kB)
Collecting ruamel.yaml<0.16,>0.15
  Using cached ruamel.yaml-0.15.100.tar.gz (318 kB)
  Preparing metadata (setup.py) ... done
Collecting psycopg2-binary<3,>=2.7.4
  Using cached psycopg2_binary-2.9.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)
Collecting argcomplete<2,>=1.9.4
  Using cached argcomplete-1.12.3-py2.py3-none-any.whl (38 kB)
Collecting sparsebitfield<1,>=0.2.3
  Using cached sparsebitfield-0.2.5.tar.gz (101 kB)
  Preparing metadata (setup.py) ... done
Collecting cerberus<2,>=1.2
  Using cached Cerberus-1.3.5-py3-none-any.whl (30 kB)
Collecting pycryptodome<4,>=3.6.1
  Using cached pycryptodome-3.19.0-cp35-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)
Collecting pyparsing<3,>=2.3.0
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting semantic-version<3,>=2.8.1
  Using cached semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)
Collecting dateparser<2,>=1.1.1
  Using cached dateparser-1.1.8-py2.py3-none-any.whl (293 kB)
Collecting structlog>=19.1.0
  Using cached structlog-23.1.0-py3-none-any.whl (62 kB)
Collecting colorama<1,>=0.4.1
  Using cached colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting diskcache>=3.0.6
  Using cached diskcache-5.6.3-py3-none-any.whl (45 kB)
Collecting attrs<22,>=21.4.0
  Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting zstandard>=0.9.0
  Using cached zstandard-0.21.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB)
Collecting blinker<2,>=1.4
  Using cached blinker-1.6.2-py3-none-any.whl (13 kB)
Collecting prometheus-client<1,>=0.7.0
  Using cached prometheus_client-0.17.1-py3-none-any.whl (60 kB)
Collecting boto3>=1.15.0
  Using cached boto3-1.28.52-py3-none-any.whl (135 kB)
Collecting Mako
  Using cached Mako-1.2.4-py3-none-any.whl (78 kB)
Collecting typing-extensions>=4
  Using cached typing_extensions-4.8.0-py3-none-any.whl (31 kB)
Collecting botocore<1.32.0,>=1.31.52
  Using cached botocore-1.31.52-py3-none-any.whl (11.2 MB)
Collecting jmespath<2.0.0,>=0.7.1
  Using cached jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting s3transfer<0.7.0,>=0.6.0
  Using cached s3transfer-0.6.2-py3-none-any.whl (79 kB)
Collecting pytz
  Using cached pytz-2023.3.post1-py2.py3-none-any.whl (502 kB)
Collecting regex!=2019.02.19,!=2021.8.27
  Using cached regex-2023.8.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (782 kB)
Collecting tzlocal
  Using cached tzlocal-5.0.1-py3-none-any.whl (20 kB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting sortedcontainers>=2.0.4
  Using cached sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
Collecting greenlet!=0.4.17
  Using cached greenlet-2.0.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (618 kB)
Collecting urllib3<1.27,>=1.25.4
  Using cached urllib3-1.26.16-py2.py3-none-any.whl (143 kB)
Collecting MarkupSafe>=0.9.2
  Using cached MarkupSafe-2.1.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28 kB)
Installing collected packages: sortedcontainers, ruamel.yaml, pytz, PrettyTable, cerberus, argcomplete, zstandard, urllib3, tzlocal, typing-extensions, structlog, sparsebitfield, six, setproctitle, semantic-version, regex, pyparsing, pycryptodome, psycopg2-binary, prometheus-client, MarkupSafe, jmespath, greenlet, diskcache, colorama, blinker, attrs, sqlalchemy, python-dateutil, Mako, dateparser, botocore, alembic, s3transfer, benji, boto3
  DEPRECATION: ruamel.yaml is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for ruamel.yaml ... error
  error: subprocess-exited-with-error

  × Running setup.py install for ruamel.yaml did not run successfully.
  │ exit code: 1
  ╰─> [54 lines of output]
      sys.argv ['/tmp/pip-install-9ah1kbt3/ruamel-yaml_f55e579de8464d4ea90be0c00c5a0c45/setup.py', 'install', '--record', '/tmp/pip-record-85cpjnxd/install-record.txt', '--single-version-externally-managed', '--compile', '--install-headers', '/usr/local/benji/include/site/python3.11/ruamel.yaml']
      test compiling test_ruamel_yaml
      /usr/local/benji/lib/python3.11/site-packages/setuptools/dist.py:285: SetuptoolsDeprecationWarning: The namespace_packages parameter is deprecated, consider using implicit namespaces instead (PEP 420).
        warnings.warn(msg, SetuptoolsDeprecationWarning)
      running install
      /usr/local/benji/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/ruamel
      copying .ruamel/__init__.py -> build/lib.linux-x86_64-cpython-311/ruamel
      creating build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./compat.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./configobjwalker.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./representer.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./scalarfloat.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./cyaml.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./__init__.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./reader.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./scalarbool.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./scanner.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./scalarint.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./anchor.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./scalarstring.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./events.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./loader.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./parser.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./comments.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./dumper.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./tokens.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./emitter.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./nodes.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./error.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./main.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./serializer.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./composer.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./timestamp.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./resolver.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./util.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./constructor.py -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./LICENSE -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      copying ./py.typed -> build/lib.linux-x86_64-cpython-311/ruamel/yaml
      running build_ext
      building '_ruamel_yaml' extension
      creating build/temp.linux-x86_64-cpython-311
      creating build/temp.linux-x86_64-cpython-311/ext
      x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/usr/local/benji/include -I/usr/include/python3.11 -c ext/_ruamel_yaml.c -o build/temp.linux-x86_64-cpython-311/ext/_ruamel_yaml.o
      ext/_ruamel_yaml.c:196:12: fatal error: longintrepr.h: No such file or directory
        196 |   #include "longintrepr.h"
            |            ^~~~~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> ruamel.yaml

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

[Gert-P]

I have managed to get Benji to work on Proxmox V8 by installing Python 3.9, creating the a Virtual Environment and installing Benji using pip. I also had to copy the python3-rados and python3-rbd modules from proxmox 7 to the Virtual Environment as the ones provided by Proxmox v8 only supports Python 3.11. Now benji is working as before.

[wech71]

The problem is the ruamel.yaml verion. In the setup.py the version check was changed to 'ruamel.yaml>=0.17,<0.18' which installs ruamel.yaml-0.16.13. Unfortunately the pypi version is 0.17 is behind and tries to use an older ruamel.yaml version which does not work with python 3.11.

For now I have set up an python 3.11 venv and pulled the source and use a helper "benji" script to load the code from my checkout path and installed the dependencies in the venv.

pip3 install 'PrettyTable>=0.7.2,<1'
pip3 install 'sqlalchemy>=2.0.7,<3'
pip3 install 'setproctitle>=1.1.8,<2'
pip3 install 'python-dateutil>=2.6.0,<3'
pip3 install 'alembic>=1.10.2,<2'
pip3 install 'ruamel.yaml>=0.17,<0.18'
pip3 install 'psycopg2-binary>=2.7.4,<3'
pip3 install 'argcomplete>=1.9.4,<2'
pip3 install 'sparsebitfield>=0.2.5,<1'
pip3 install 'cerberus>=1.2,<2'
pip3 install 'pycryptodome>=3.6.1,<4'
pip3 install 'pyparsing>=2.3.0,<3'
pip3 install 'semantic_version>=2.8.1,<3'
pip3 install 'dateparser>=1.1.1,<2'
pip3 install 'structlog>=19.1.0'
pip3 install 'colorama>=0.4.1,<1'
pip3 install 'diskcache>=3.0.6'
pip3 install 'attrs >=21.4.0,<22'

[wech71]

For the sake of completeness what I did:

a) check out benji to shared dir on cephfs

git clone https://github.com/elemental-lf/benji.git /mnt/pve/cephfs/benji-src

b) Replaced /usr/local/bin/benji with:

source /usr/local/benji/bin/activate
python3 /mnt/pve/cephfs/benji-python3.11 $*

c) chmod +x /usr/local/bin/benji

d) Created modified benji-entrypoint-script
/mnt/pve/cephfs/benji-python3.11:

#!/usr/bin/python3
# -*- coding: utf-8 -*-
import re
import sys

sys.path.append('/mnt/pve/cephfs/benji-src/src')

from benji.scripts.benji import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

e) make python-rados/rdb modules available to venv:

ln -s /usr/lib/python3/dist-packages/rados.cpython-311-x86_64-linux-gnu.so /usr/local/benji/lib/python3.11/site-packages/rados.cpython-311-x86_64-linux-gnu.so;
ln -s /usr/lib/python3/dist-packages/rbd.cpython-311-x86_64-linux-gnu.so /usr/local/benji/lib/python3.11/site-packages/rbd.cpython-311-x86_64-linux-gnu.so;

[wech71]

@elemental-lf Could you please update the official benji pypi package (https://pypi.org/project/benji/).

Proxmox 8 upgrades python to 3.11, but the current benji pypi version 0.17.0, uses an older ryamel dependency which does not compilies on python 3.11.

The problem is alredy fixed in the benji master branch, but it currently can't any longer be installed or updated trough pip.