[BUG] distutils is deprecated and slated for removal in python3.12

[bryceml]

Description
from setup.py on py3.10

DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives

We need to update our setup.py to not use distutils by the time python3.12 is out.

[waynew]

Started to dig into this, and it's somewhat promising! Not great or perfect, but promising.

I'm playing around on my fork https://github.com/waynew/salt/tree/distutils-removal to see what does what.

I built a setup.cfg based off some limited information from the setup.py., ran pip wheel . -w dist/ and was able to build a wheel, install it on arch linux, and run a very limited test: started salt-master & salt-minion and ran salt-key -Ay, salt \* test.version and salt \* pkg.install ed and they all worked without a hitch.

This is obviously a very limited subset of functionality, but the fact that nothing broke or went horribly wrong is a good sign. One thing that is definitely going to need changes, though, is the bootstrap script, since right now it runs setup.py.

In theory we could just do the setup.py version (or perhaps also allow setup.py) but I think taking this opportunity to slim some things down would be a good idea 🤔

Of course, the tiamat builds might make the bootstrap script less of an issue. I'm also not familiar with the rest of the packaging, like rpms and debs. 🤔

[nicholasmhughes]

This also affects a few modules as well...

$ grep -rIl distutils *

cloud/deploy/bootstrap-salt.sh
modules/puppet.py
modules/virtualenv_mod.py
utils/pkg/win.py
utils/versions.py

...and then utils/versions.py extends the LooseVersion and StrictVersion classes, which is used all over the place:

$ grep -ErIl 'StrictVersion|LooseVersion' *

cloud/clouds/cloudstack.py
cloud/clouds/openstack.py
cloud/clouds/dimensiondata.py
cloud/clouds/gce.py
cloud/clouds/profitbricks.py
fileserver/azurefs.py
modules/libcloud_storage.py
modules/mongodb.py
modules/pacmanpkg.py
modules/libcloud_loadbalancer.py
modules/mac_service.py
modules/puppet.py
modules/mac_assistive.py
modules/launchctl_service.py
modules/opkg.py
modules/npm.py
modules/kernelpkg_linux_apt.py
modules/postgres.py
modules/zypperpkg.py
modules/rabbitmq.py
modules/libcloud_compute.py
modules/chocolatey.py
modules/kernelpkg_linux_yum.py
modules/yumpkg.py
modules/zabbix.py
modules/lxc.py
modules/bower.py
modules/rpm_lowpkg.py
modules/file.py
modules/git.py
modules/libcloud_dns.py
modules/travisci.py
modules/lxd.py
modules/tls.py
modules/gpg.py
modules/win_pkg.py
netapi/rest_tornado/__init__.py
netapi/rest_cherrypy/__init__.py
pillar/ec2_pillar.py
returners/mongo_future_return.py
returners/xmpp_return.py
returners/mongo_return.py
states/mac_assistive.py
states/pkg.py
states/boto3_elasticsearch.py
states/git.py
utils/openstack/nova.py
utils/jinja.py
utils/pkg/win.py
utils/versions.py
utils/win_network.py
utils/gitfs.py
utils/templates.py
utils/network.py
utils/win_dotnet.py

[twangboy]

We're starting to see this with every salt command:

PS C:\Users\vagrant> salt-call --version
C:\Program Files\Salt Project\Salt\bin\lib\site-packages\_distutils_hack\__init__.py:33: UserWarning: Setuptools is repl
acing distutils.
  warnings.warn("Setuptools is replacing distutils.")
salt-call 3005rc1

[twangboy]

This is in Master

[dmurphy18]

Running into issue on replacing distutils.sysconfig.get_python_lib() on Debian / Ubuntu since this returns the path

>>> distutils.sysconfig.get_python_lib()
'/usr/lib/python3/dist-packages'

and is discussed here in Python bugs https://bugs.python.org/issue41282#msg393021

proposed replacement is as follows:

>>> sysconfig.get_path("purelib")
'/usr/lib/python3.9/site-packages'

This could affect previously installed packages, but we are using onedir moving forward, however those with forks of Salt Project will be affect, for example, the Debian fork of Salt.

[dmurphy18]

run into issue with generating versions for Salt, setup.py wrote out salt/_versions.py when generating sdist, see write_salt_version etc.
However the versions generated differ from that using setuptools-scm, for example: on the current master branch

david@david-XPS-15-9570:~/devcode/dgm_salt_master/salt$ python3 -m setuptools_scm 
Warning: could not use pyproject.toml, using default configuration.
 Reason: /home/david/devcode/dgm_salt_master/salt/pyproject.toml does not contain a tool.setuptools_scm section.
/home/david/.pyenv/versions/3.8.6/lib/python3.8/site-packages/setuptools_scm/config.py:61: UserWarning: relative_to is expected to be a file, its the directory '.'
assuming the parent directory was passed
  warnings.warn(
3005.2.dev1490+gb728d0e0e4
david@david-XPS-15-9570:~/devcode/dgm_salt_master/salt$ python3 setup.py --version
3005.1+1490.gb728d0e0e4
3005.1+1490.gb728d0e0e4

and untaring a python3 setup.py sdist generated tar ball

tar -xzvf salt-3005.1+1490.gb728d0e0e4.tar.gz
.
.
david@david-XPS-15-9570:~/devcode/dgm_salt_master/salt/dist$ cat salt-3005.1+1490.gb728d0e0e4/salt/_version.py 
# This file was auto-generated by salt's setup

from salt.version import SaltStackVersion

__saltstack_version__ = SaltStackVersion(3005, 1, None, 0, '', 0, 1490, 'gb728d0e0e4')
david@david-XPS-15-9570:~/devcode/dgm_salt_master/salt/dist$

so need to reconcil the setuptools-scm 3005.2 ??? and why it is 3005.2 vs the current 3005.1

Need to ensure that no matter the method setup.py or setuptools-scm, the same version is reported for Salt.

[dmurphy18]

Agreed work for now keeping setup.py since work-flows and usage patterns (was always the plan) but need to resolve how version is handled, especially since want to be able to release a nightly build that has passed all tests etc. as v300x.y.z without having to rebuild anything, which currently can not do with distutils or setuptools approaches.

[dmurphy18]

Problem with 3006.0 is due to changes in salt/utils/version.py, changes to StrictVersion or LooseVersion cause the problem to occur, noting that a pip3 list | grep salt reuturns 3006.0, but salt-call --local test.version returns the 3005.1+1545.g776c30659f, with distutils.version but those agree with 3005.1+1545.g776c30659f.

The PyPI looseversion is not a direct compatible direct replacement for distutils.version.LooseVersion as it stated.

[dmurphy18]

Ran into the cause of the issue of version 3006.0 appearing and is due to an ImportError in salt/version.py.
If you get an ImportError, the code drops through to attempt to __discover_version, see https://github.com/saltstack/salt/blob/master/salt/version.py#L645-L656
which attempts to discover the version from salt/version.py and it returns 3006.0 (basically the current_release - imho poorly named since it is really the next release, as the current release is actually 3005.1, master branch current version is 3005.1+1546.g974558464a)

With Python recommending using packaging.version instead of distutils.version, and thus import from packaging, results in an ImportError with pip install

  49     File "/tmp/pip-req-build-pbnkyin6/salt/utils/versions.py", line 44, in <module>
  50       from packaging.version import InvalidVersion
  51   ModuleNotFoundError: No module named 'packaging'

This implies needing to add build-system to the pyproject.toml file, such that pip will install the needed packages, packaging and looseversion before attempting to install the rest of Salt (note the problem occurs during pip's 'Getting requirements to build wheel: started', so there is not a chance to fuly resolve all the requirements for Salt.
The pyproject.toml additions:

[build-system]
requires = ["setuptools", "wheel", "packaging", "looseversion"]
build-backend = "setuptools.build_meta"

However this result in the following ModuleLoadError

   9 Processing ./salt-3005.1+1545.g776c30659f.tar.gz
  10   Created temporary directory: /tmp/pip-req-build-k51mbuzs
  11   Added file:///home/david/tmp_salt/salt-3005.1%2B1545.g776c30659f.tar.gz to build tracker '/tmp/pip-build-tracker-tlooebmj'
  12   Created temporary directory: /tmp/pip-build-env-jkrp9_am
  13   Installing build dependencies: started
  14   Running command pip subprocess to install build dependencies
  15   Using pip 22.3.1 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)
  16   Collecting setuptools
  17     Using cached setuptools-65.6.3-py3-none-any.whl (1.2 MB)
  18   Collecting wheel
  19     Using cached wheel-0.38.4-py3-none-any.whl (36 kB)
  20   Collecting packaging 
  21     Downloading packaging-22.0-py3-none-any.whl (42 kB)
  22        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.6/42.6 kB 740.9 kB/s eta 0:00:00
  23   Collecting looseversion 
  24     Using cached looseversion-1.0.2-py3-none-any.whl (6.4 kB)
  25   Installing collected packages: wheel, setuptools, packaging, looseversion
  26     Creating /tmp/pip-build-env-jkrp9_am/overlay/bin 
  27     changing mode of /tmp/pip-build-env-jkrp9_am/overlay/bin/wheel to 755
  28   ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
  29   launchpadlib 1.10.13 requires testresources, which is not installed.
  30   Successfully installed looseversion-1.0.2 packaging-22.0 setuptools-65.6.3 wheel-0.38.4
  31   WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/w     arnings/venv
  32   Installing build dependencies: finished with status 'done'
  33   Getting requirements to build wheel: started
  34   Running command Getting requirements to build wheel
.
.
  43     File "/tmp/pip-req-build-k51mbuzs/salt/version.py", line 684, in __get_version
  44       from salt._version import __saltstack_version__  # pylint: disable=E0611,F0401
  45   ModuleNotFoundError: No module named 'salt'

This problem occurs even if you remove any changes in salt/utils/version.py, that is back to using distutils.version. The presence of the build-system commands in pyproject.toml causes the issue. This may be due to pyproject.toml in its current state is not fully fleshed out for use, for example: nothing defining project salt.

Solution for now is to utilize setuptools._distutils.version for now, and allow time to fully research pyproject.toml issues and fully flesh it out for use with Salt. Note Setuptools links still shows pyproject.toml usage as BETA as of 2022-12-15 , url https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#beta

Support for adding build configuration options via the [tool.setuptools] table in the pyproject.toml file is still in beta stage.