[BUG] TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero' with setuptools 71.x

[andy-maier]

setuptools version

71.0.4

Python version

3.12

OS

macOS

Additional environment information

This is only happening since setuptools 71.0.0.

Description

I tried installing my setup.py based project (https://github.com/zhmcclient/zhmccli) from the checked-out repo clone and setuptools failed as follows:

$ python -m pip install --upgrade --upgrade-strategy eager -e .
Obtaining file:///Users/maiera/Projects/zhmcclient/repos/zhmccli
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [46 lines of output]
      running egg_info
      creating /private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/pip-pip-egg-info-bknai7ob/zhmccli.egg-info
      writing /private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/pip-pip-egg-info-bknai7ob/zhmccli.egg-info/PKG-INFO
      writing dependency_links to /private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/pip-pip-egg-info-bknai7ob/zhmccli.egg-info/dependency_links.txt
      writing entry points to /private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/pip-pip-egg-info-bknai7ob/zhmccli.egg-info/entry_points.txt
      writing requirements to /private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/pip-pip-egg-info-bknai7ob/zhmccli.egg-info/requires.txt
      writing top-level names to /private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/pip-pip-egg-info-bknai7ob/zhmccli.egg-info/top_level.txt
      writing manifest file '/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/pip-pip-egg-info-bknai7ob/zhmccli.egg-info/SOURCES.txt'
      reading manifest file '/private/var/folders/lh/v0_07k9d7dbfqdytfzzxks3r0000gn/T/pip-pip-egg-info-bknai7ob/zhmccli.egg-info/SOURCES.txt'
      adding license file 'LICENSE'
      adding license file 'AUTHORS.md'
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/Users/maiera/Projects/zhmcclient/repos/zhmccli/setup.py", line 81, in <module>
          setuptools.setup(
        File "/Users/maiera/virtualenvs/zhmccli312/lib/python3.12/site-packages/setuptools/__init__.py", line 108, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/maiera/virtualenvs/zhmccli312/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/Users/maiera/virtualenvs/zhmccli312/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/Users/maiera/virtualenvs/zhmccli312/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 970, in run_commands
          self.run_command(cmd)
        File "/Users/maiera/virtualenvs/zhmccli312/lib/python3.12/site-packages/setuptools/dist.py", line 956, in run_command
          super().run_command(command)
        File "/Users/maiera/virtualenvs/zhmccli312/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 989, in run_command
          cmd_obj.run()
        File "/Users/maiera/virtualenvs/zhmccli312/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 310, in run
          self.find_sources()
        File "/Users/maiera/virtualenvs/zhmccli312/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 318, in find_sources
          mm.run()
        File "/Users/maiera/virtualenvs/zhmccli312/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 544, in run
          self.prune_file_list()
        File "/Users/maiera/virtualenvs/zhmccli312/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 610, in prune_file_list
          base_dir = self.distribution.get_fullname()
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/maiera/virtualenvs/zhmccli312/lib/python3.12/site-packages/setuptools/_core_metadata.py", line 266, in get_fullname
          return _distribution_fullname(self.get_name(), self.get_version())
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/maiera/virtualenvs/zhmccli312/lib/python3.12/site-packages/setuptools/_core_metadata.py", line 284, in _distribution_fullname
          canonicalize_version(version, strip_trailing_zero=False),
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

I took a closer look and found:

• The error shows up only since setuptools 71.0.0
• Upgrading the "packaging" package from 21.3 to 24.1 (latest) solved the error. It turns out that packaging 22.0 is the first version solving the error.

The reason I had "packaging" 21.3 installed is because some tools used in the development environment for the project (installed with make develop) used it. The dev-requirements.txt file of the project meanwhile specifies packaging>=23.2, but the virtual env was still from an earlier state of the project.

The problem is that "setuptools" seems to have a soft dependency on "packaging": It does not require it to be installed, but when it is installed it is used, e.g in its _core_metadata.py file. So "setuptools" ended up not specifying it in its package dependencies, but when it is installed, it still needs a certain minimum version, which is not ensured to be present because the package is not specified in the dependencies.

That puts the burden on the users to make sure that "packaging" when needed by other tools, has the required minimum version needed by "setuptools". Not a good situation.

Expected behavior

When "setuptools" uses "packaging" and does not specify it as a dependency, it should to make sure that it works with all versions of "packaging" that can be installed on the given Python version.

There are of course several alternatives to using "packaging" without specifying it as a dependency (vendoring it, specifying it as a dependency, not using it).

How to Reproduce

1. Have a fresh virtualenv based Python 3.12 environment. It initially has:

   Package    Version
   ---------- -------
   pip        24.1.2
   setuptools 70.0.0
   wheel      0.43.0
   

2. Clone https://github.com/zhmcclient/zhmccli and cd into its main directory
3. run python -m pip install --upgrade --upgrade-strategy eager -e .
   This works fine.
   After this, "setuptools" is still at 70.0.0 and "packaging" is not installed.
   This shows that "setuptools" 70.0.0 works without "packaging".
4. run pip install setuptools==71.0.4
   After this, "packaging" is still not installed because "setuptools" does not specify it as a dependency.
5. run python -m pip install --upgrade --upgrade-strategy eager -e .
   This works fine and shows that "setuptools" 71.0.4 works without "packaging".
6. run pip install packaging==21.3
   After this, "packaging" is at 21.3 and "setuptools" is still at 71.0.4.
7. run python -m pip install --upgrade --upgrade-strategy eager -e .
   This fails as shown above, since "packaging" is below the version needed by "setuptools".
8. run pip install packaging==22.0
9. run python -m pip install --upgrade --upgrade-strategy eager -e .
   This works fine again, since "packaging" is now at the version needed by "setuptools".

Output

The output is long, and the relevant output from the failure is shown in the description above.

[andy-maier]

Update: After opening this issue, I found that the closed issue #4483 describes the same problem.

I have to say though that the answer "this is intentionally by design" in #4483 does not satisfy me at all.

How can you use another package that you are neither vendorizing nor specifying as a dependency, and then let your users figure out that they have to install a certain minimum version of that package? In a package like setuptools that everybody has to use? So you now want a million package owners that use other tools with a dependency on "packaging" to make sure that "packaging" has the right minimum level? Does "setuptools" document that minimum level?

I checked which other packages in my project use which "packaging" version, and here is the result. This is on Python 3.12 with latest versions of all tools installed, and you see that there are plenty of tools that require "packaging" versions below 22.0, so unless a user happens to use (a version of) a tool that requires the right minimum version, they will run into this issue:

$ python -m pipdeptree -r -p packaging
Warning!!! Duplicate package metadata found:
"/Users/maiera/Projects/zhmcclient/repos/zhmccli"
  zhmccli                          1.12.0.dev1      (using 1.12.0.dev1, "/Users/maiera/Projects/zhmcclient/repos/zhmccli")
NOTE: This warning isn't a failure warning.
------------------------------------------------------------------------
packaging==24.1
├── build==1.2.1 [requires: packaging>=19.1]
├── dparse==0.6.4b0 [requires: packaging]
│   ├── safety==3.2.3 [requires: dparse>=0.6.4b0]
│   └── safety-schemas==0.0.2 [requires: dparse>=0.6.4b0]
│       └── safety==3.2.3 [requires: safety-schemas>=0.0.2]
├── marshmallow==3.21.3 [requires: packaging>=17.0]
│   └── safety==3.2.3 [requires: marshmallow>=3.15.0]
├── pip-check-reqs==2.5.3 [requires: packaging>=20.5]
├── pipdeptree==2.23.1 [requires: packaging>=23.1]
├── pyinstaller-hooks-contrib==2024.1 [requires: packaging>=22.0]
├── pyproject-api==1.7.1 [requires: packaging>=24.1]
│   └── tox==4.16.0 [requires: pyproject-api>=1.7.1]
├── pytest==8.3.1 [requires: packaging]
│   └── pytest-cov==5.0.0 [requires: pytest>=4.6]
├── safety==3.2.3 [requires: packaging>=21.0]
├── safety-schemas==0.0.2 [requires: packaging>=21.0]
│   └── safety==3.2.3 [requires: safety-schemas>=0.0.2]
├── Sphinx==7.4.7 [requires: packaging>=23.0]
│   ├── autodocsumm==0.2.12 [requires: Sphinx>=2.2,<8.0]
│   ├── sphinx-git==11.0.0 [requires: Sphinx]
│   ├── sphinx-rtd-theme==2.0.0 [requires: Sphinx>=5,<8]
│   ├── sphinxcontrib-jquery==4.1 [requires: Sphinx>=1.8]
│   │   └── sphinx-rtd-theme==2.0.0 [requires: sphinxcontrib-jquery>=4,<5]
│   └── sphinxcontrib-websupport==1.2.7 [requires: Sphinx>=5]
└── tox==4.16.0 [requires: packaging>=24.1]

[jaraco]

Please feel free to rephrase your question in light of the context in the duplicate issue (if it's still relevant in that context).