[DOC] PEP 621 - pyproject.toml vs setup.cfg

[vancromy]

Summary

Hello,

As a newcomer to the python packaging world I was redirected from python.org to the setuptools documentation because python recommended it as the first choice for packaging. I had a quick read through the documentation and see that there are three different files for specifying project metadata/settings setup.py (for legacy projects - to be deprecated and kept to a bare minimum), setup.cfg (as I understood from the docs, some sort of successor of setup.py but with no code execution) and finally pyproject.toml. On the Configuring setuptools using pyproject.toml files page, the documentation mentions that since PEP 621 the python community has adopted the pyproject.toml as the go-to config file for specifying distribution/project metadata/settings. I therefore am left a bit confused as to what the use case is for the setup.cfg file and I did not feel like the documentation properly explained this. There is a separate page for Configuring setuptools with setup.cfg files but it doesn't address the overlap/difference between the two. I was therefore hoping that the experts in this community could clarify this and potentially have a separate page in the documentation (e.g. setup.cfg vs pyproject.toml) describing:

• The difference between the two and which use cases they serve
• Whether they should coexist or whether newer and older projects should aim to migrate to one of them as a result of PEP 621
• A much clearer (black and white) recommendation on what the ideal or python recommended way is of defining a project's configuration

I had a quick scan and search of the GitHub issues to see if this had been brought up before but got no hits on my search so apologies if this is a dupe.

Note: I could have made this a discussion but specifically chose to opt for an issue because I felt an improvement to the docs was necessary for clarity's sake.

Looking forward to hearing from the experts!
YVC

OS / Environment

N/A

Additional Information

For newbies like myself and potentially also intermediate/advanced developers, this clarity would hopefully ensure we are all going in the same direction in terms of the future of python packaging. :)

Code of Conduct

• I agree to follow the PSF Code of Conduct

[mcrumiller]

The discussion over at #3214 may be helpful.

I've since moved my projects to a pure-pyproject.toml approach in order to stay on the bleeding edge of project packaging, but there is massive drawback: building packages in editable mode is an awful, awful experience. Currently, with pure pyproject.toml, the only way to allow editable mode is to use pip install -e . --config-settings editable_mode=strict, which typically creates a new folder of symlinked files that it treats as the actual package source. The issue for me (which is probably unrelated to setuptools) is that these symlinks constantly break in Windows with no warning, which means a dangerous work environment where I'm not sure which files have which edits.

I'd love to hear some more experienced developers chime in though. Is pure-pyproject.toml still in its infancy and not recommended?

[jenshnielsen]

@mcrumiller I am curios in which sense the is a awful, awful experience I have been using it for some time. The one major issue that I am aware of is #3518 which breaks IDE / static checker integration.
The recommended way to work around this issue for now seems to be to use the legacy behaviour

The new editable install path however, depends on the implementation of pep660 and not pep621 as far as I am aware.
You will get pep660 behaviour if you define a build back end in pyproject.toml but should be independent of whether you put the rest of your config in setup.py/setup.cfg or pyporject.toml

E.g. as long as you have

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

or similar in your pyproject.toml file

[mcrumiller]

@jenshnielsen I was a little gushy in my over-use of the word awful, I apologize. Here are my main gripes:

• legacy behavior requires setup.py (or I haven't figured out how to use it otherwise)
• Once installed, VS Code does not detect breakpoints set in the main source files, only the files set in the build/__editable__* directories.
• Soft links appear to break randomly without any notification. This combined with previous bullet point means 1) you should never work on files in the build/__editable__* directories, but 2) you can't set breakpoints in the same files you're working on, making the process pretty clunky.
• Performing a version bump in __init__.py now requires a reinstall of the package (not a big issue).

I admit I am quite the noob on build backends so I'm not here to complain but rather see if others have gone through and resolved any of my issues outlined above.

[jenshnielsen]

Thanks, those sounds like they are related to #3518
You should be able to get the legacy behavior with

pip install -e . --config-settings editable_mode=compat

As far as I understand that does not depend on the existence of the setup.py file

[mcrumiller]

@jenshnielsen I swear I tried compat mode in the past and pylance failed to recognize editable installs--but now it works! Thank you so much.

[mcrumiller]

Spoke too soon: compat mode doesn't seem to work at all:

(.venv_brr) C:\Projects>pip install -e ./python-toolbox --config-settings editable_mode=compat
<installation output>
Successfully installed python-toolbox-1.1.5

(.venv_brr) C:\Projects>python -c "import toolbox"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'toolbox'

I tried using SETUPTOOLS_ENABLE_FEATURES="legacy-editable" as suggested here but I get the following error:

ERROR: Project file:///C:/Projects/python-toolbox has a 'pyproject.toml' and its build backend is missing the
'build_editable' hook. Since it does not have a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable
mode. Consider using a build backend that supports PEP 660.

Any suggestions?

[jenshnielsen]

I assume that python-toolbox is https://github.com/cool-RR/python_toolbox . It seems like that is a pure setup.py project? To me it looks like compat mode is not working correctly in that case. Not sure it that is a bug but I would think so? (I am not a setuptool dev just a user)

[mcrumiller]

Sorry nope, it's my own local package located in C:\Projects\python-toolbox

[jenshnielsen]

Ok could you share the pyproject.toml file from that project?

[abravalheri]

ERROR: Project file:///C:/Projects/python-toolbox has a 'pyproject.toml' and its build backend is missing the
'build_editable' hook. Since it does not have a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable
mode. Consider using a build backend that supports PEP 660.

Any suggestions?

Update the version of pip/setuptools you are using/have installed?

[mcrumiller]

Here's the gist of it:

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
authors = [
    { name = "me", email = "[email protected]" },
]
name = "python-toolbox"
description = "Python toolbox"
dynamic = ["version"]
dependencies = [
    "my_dependencies"
]

[project.scripts]
script_name = "module:function"

[tool.setuptools]
include-package-data = true
packages = [
    "toolbox",
    "toolbox.subpackages",
    "my_subpackages"
]

[tool.setuptools.package-dir]
toolbox = "src/python/toolbox"

[tool.setuptools.package-data]
    "toolbox" = "my_resource"
    "toolbox.subpackage" = "subpackage_resource"

[tool.setuptools.dynamic]
version = { attr = "toolbox.__version__" }
readme = { file = ["README.md"] }

My project is setup like the following:

python-toolbox
 |--src
     |--python
         |--toolbox
             |--__init__.py
             |--other files

[vancromy]

@mcrumiller @jenshnielsen Happy to see you guys hash out some kinks with editable installs using pyproject.toml but that seems like it should perhaps be in a dedicated issue specifically for addressing editable install issues when using a pyproject.toml file.

I'm afraid this discussion may lead to the focus getting shifted away from the core issue which was a call for more clarity and better guidance on which type of config file to go for when setting up or updating a Python package to meet today's packaging standards.

I do thank you for pointing me to #3214 because I learnt that setup.cfg files may become obsolete. I think that just strengthens my case for requiring a clearer stance in the docs as to what developers should be doing.

[jenshnielsen]

Agreed, @mcrumiller perhaps it makes more sense for you to create a new bug for the issue that you are seeing with the information from here

[mcrumiller]

Sorry to derail the conversation @vancromy. If anyone is interested I've opened #3767 for my own issue.

[abravalheri]

@vancromy, I am sorry to disappoint, but I don't think you are going to find objective technical reasons for using one or the other.

If I am not wrong the setup.cfg format evolved first (even from outside setuptools). The motivation why pyproject.toml exists is kind of external to the setuptools implementation itself, and more related to standardisation efforts.

Using pyproject.toml, however, imposes constraints (specially around dynamic) to the build. Some users are not comfortable with these constraints, so I don't think there should be a "black and white" recommendation.

If you are OK with these constraints, my personal recommendation (and I don't speak for setuptools here) is:

1. Use the src-layout that is good for autodiscovery and editable installation. That will make your life easier
2. Use pyproject.toml and its [project] table to add metadata. If you are using a standard src-layout, autodiscovery will do most of the work for you and there will be no need to configure [tool.setuptools].
3. Use setuptools-scm, so you are sure all the files under version control are included in the final distribution. This also helps with versioning - you can make the git tags the single source of truth for versions.
4. If you need to add C/Cython extensions add a minimal setup.py file (alongside pyproject.toml, e.g.:

   from setuptools import setup, Extension
   setup(ext_modules=[Extension(...), ...])

[jenshnielsen]

@abravalheri is there any update on the beta status of [tools.setuptools] in pyproject.toml. I have a few projects that uses pyproject.toml but I have kept around setup.cfg for the setuptools specific config. This is mainly for stuff like zip_safe and package_data since I am not using setuptools-scm.
It would be very nice to be able to remove the setup.cfg file all together for these projects

[impaktor]

Any update?

[ptth222]

@abravalheri This is exactly what I ended up doing except for part 4. I was able to get Cython working without a setup.py. It still requires a .py file, but it doesn't rely on setup.py being special. This essentially makes it so I have a pure pyproject.toml solution which is what I think the future is going to be. For anyone interested in an example you can checkout my repo https://github.com/MoseleyBioinformaticsLab/MESSES.