uv should provide a build backend

[chrisrodrigue]

uv is a fantastic tool that is ahead of it's time. In the same vein as ruff, it is bundling many capabilities that Python developers need into a single tool. It currently provides the capabilities of pip, pip-tools, and virtualenv in one convenient binary.

Python can't build out of the box

Python 3.12 has removed setuptools and wheel which means that a standard Python installation now has no way to build or package a project from pyproject.toml. This is a glaring flaw with Python, which is supposed to be "batteries included." As of Python 3.12, a user is unable to perform pip install -e . or pip install . on a local project without installing third party packages. This means that in an offline environment without access to PyPi, a developer is dead in the water and cannot install local projects from source.

uv can fix this

I propose that uv expand its capabilities to support all pip commands, including pip install [-e] . and pip wheel. The default behavior of pip install [-e] . in a project without a build backend specified in pyproject.toml is to utilize setuptools to build the project, which uv could certainly do. It could also utilize more modern, PEP-compliant build backends such as pdm-backend, flit, or hatchling, and editables for an editable build backend (which I believe pdm-backend can be configured to use).

I think that if uv could natively build projects from source, it would be a game changer.

[potiuk]

I think that if uv could natively build projects from source, it would be a game changer.

I personally believe that this is pretty much against the whole idea of modern approach and splitting the backend vs. frontend responsibilities. The idea (philosophically) behind the backend/frontend split is that the maintainers of the project (via pyproject.toml) choose the backend that should be used to build their tool, while the user installing Python project is free to choose whatever fronted they prefer. This is IMHO a gamechanger in the python packaging and we should rather see a stronger push in that direction than weakening it. And I think it's not going to go back - because more and more projects will revert to use pyproject.toml and backend specification of build environment. In case of Airflow for example - as of December there is no way to install airlfow "natively" without actually installing hatchling and build environment. But we do not give the option to the frontend. You MUST use hatchling in specified version and few other dependencies specified in specific versions to build airflow. Full stop. UV won't be able to make their own choices (It will be able to choose a way how to create such an environment but not to choose what should be used to build airlfow).

But also maybe my understanding of it is wrong and maybe you are proposing something different than I understand.

BTW. I do not think access to PyPI is needed to build a project with build backend. The frontend my still choose any mechanism (including private repos if needed) to install build environment, no PyPI is needed for it, the only requirement is that pyproject.toml specifies the environment.

[chrisrodrigue]

@potiuk

I agree and think the backend and frontend specifications should be separate, I am merely suggesting that uv could provide a build capability similar to its virtualenv capability (uv venv), such that building can be possible for users that don't have one, perhaps via Project API such as uv install. I like the pdm approach of providing its own build backend (pdm-backend) and installing projects as editable by default inside the automatically managed .venv of the project when a user does pdm install.

You do need access to PyPi or some repo hosting the build backend to build a project. Python does not include setuptools or any other build backend in the latest distributions (it used to include at least setuptools). It downloads the build backend specified in the pyproject.toml (or setuptools if none specified) in an isolated environment and uses it to build python projects from source. If --no-build-isolation is specified, it expects the build-backend to already be available in the current system/virtual environment in order to build.

IMO, pip install [-e] . is broken out of the box because it relies on third party dependencies which may not be accessible. In the stdlib we get argument parsing, unit testing, logging, and other amenities, but we can't perform the most fundamental action in the software development process. Without the capability to build from source out of the box, Python is hamstringed and can only run the most rudimentary scripts, leaving users to resort to ugly PYTHONPATH/sys.path hacks to get their packages/subpackages/modules found by the interpreter.

[potiuk]

MO, pip install [-e] . is broken out of the box because it relies on third party dependencies which may not be accessible. In the stdlib we get argument parsing, unit testing, logging, and other amenities, but we can't perform the most fundamental action in the software development process. Without the capability to build from source out of the box, Python is hamstringed and can only run the most rudimentary scripts, leaving users to resort to ugly PYTHONPATH/sys.path hacks to get their packages/subpackages/modules found by the interpreter.

But this is where the whole packaging for Python is heading. The PEPs of packaging precisely specify this is the direction and if project maintainers choose so, unless you manually (like conda) maintain your build recipes for all the packages our there, you won't be able to build project "natively".

Just to give you example of Airflow. Without hatchling and build hooks (implemented in hatch_build.py you are not even able to know what Airflow dependencies are, because "requirements", "optional-requirements" are declared as dynamic fields - and they don't even have (as mandated by the right PEP) specification of those dependencies in pyproject.toml. And we have no setup.py either any more.

The only way to find out what dependencies Airflow needs for editable build, or to build a wheel package is to get the right version of hatchling and get the frontend execute the build_hook - the build hook returrns such dependencies dynamically. You can see it yourself here https://github.com/apache/airflow/blob/main/pyproject.toml -> there is no way to build airflow from sources in current main without actually installing those packages:

    "GitPython==3.1.43",
    "gitdb==4.0.11",
    "hatchling==1.24.2",
    "packaging==24.0",
    "pathspec==0.12.1",
    "pluggy==1.5.0",
    "smmap==5.0.1",
    "tomli==2.0.1; python_version < '3.11'",
    "trove-classifiers==2024.5.22",

And letting hatchling invoke hatch_build.py. I am not sure what you mean by "native" installation - but you won't be able to install airflow differently.

And I think - personally (though I was not part of it) - that the decisions made by the packaging team were pretty sound and smart, and they deliberately left the decision for maintainers of a project to choose the right backend packages needed (and set of 3rd-party tools) and all frontends have no choice but to follow it. I understand you might have different opinion, but here - the process of Python Software Foundation and Packaging team is not an opinion - they have authoritative power to decide it by voting and PEP approval. And the only way to change it is to get another PEP approved.

Here is the list of those PEPs (and I am actually quite happy Airflow after 10 years finally migrated out of setuptools and setup.py by following those standards as finally the tooling - including the modern backends you mentioned support it for sufficently long time):

• PEP-440 Version Identification and Dependency Specification
• PEP-517 A build-system independent format for source trees
• PEP-518 Specifying Minimum Build System Requirements for Python
• PEP-561 Distributing and Packaging Type Information
• PEP-621 Storing project metadata in pyproject.toml
• PEP-660 Editable installs for pyproject.toml based builds (wheel based)
• PEP-685 Comparison of extra names for optional distribution

[potiuk]

And BTW. if uv provides a build backend, you will still be able to choose it when you maintain your project - but it will also be a 3rd-party dependency :)

Someone installing your project with any frontend will have to download and install according to your specification in pyproject.toml. Similarly as hatch/hatchling pair that are separate. Even if you use hatch to install airlfow, it has to anyhow download hatchling in the version specified by the maintainer of the project you are installing and use it to build the package.

[chrisrodrigue]

It's interesting that setuptools remains the "blessed" build backend that pip defaults to use when no backend is specified in the pyproject.toml. Rather than bake setuptools into the stdlib, it silently forces the download and installation of it as an unauthorized third-party build dependency. This also seems to violate the principle of "explicit is better than implicit."

One of the nice things about Astral tools like uv and ruff is that they do not bloat your package/library when you use them, and bundle the capabilities of many tools into single static binaries.

If you use uv specifically for its virtual environment capability, it's a single dependency. Conversely, if you were to use virtualenv, you've now just pulled in 3 more transitive dependencies ( distlib, filelock, platformdirs). Similarly, instead of using pylint, flake8, pyupgrade, black, isort, and all of their dependencies, you only need ruff. This has far reaching implications for companies abiding by Software Bill of Materials (SBOMs) that need to manually vet each piece of FOSS in their software toolchains.

I think uv could provide a build backend that could be specified in pyproject.toml and used as the default when no backend is specified. It doesn't necessarily have to be separate from the main uv binary, since the uv binary could support the build capability as builtin uv/uv pip commands (uv install, uv pip install -e .).

[build-system]
requires = ["uv"]
build-backend = "uv.build"

[potiuk]

I think uv could provide a build backend that could be specified in pyproject.toml and used as the default when no backend is specified. It doesn't necessarily have to be separate from the main uv binary, since the uv binary could support the build capability as builtin uv/uv pip commands (uv install, uv pip install -e .).

Well. There are always trade-offs and what you see from your position might be important for you, might not be important for others and the other way. For example - If (like we did with airflow being popular package) you had gone through some of the pains where new releases of setuptools suddently started breaking packages being built - because deliberately or accidentally breaking compatibilities and suddenly being flooded by 100s of your users having problem with installing your package without you doing anything you'd understand that bundling the way how things are run with specific version of frontend is a good idea when you have even moderately big package.

That's what I love that we as maintainers can choose and "lock" the backend to the tools and version of our choice - rather than relying that the version of the tool that our users choose will behave consistently over the years. That was a very smart choice of packaging team based on actual learnings from their millions of users over many years, and while I often criticized their choices in the past, I came to understanding that it's my vision that is short-sighted and limited - I learned a bit of empathy.

I'd strongly recommend a bit more reading and understanding what they were (and still do) cooking there. Python actually deliberately removeed setuptools in order to drive more the adoption of what's being developed as packaging standards (and that's really smart plan that was laid out years ago and is meticulously and consistently, step-by-step put in motion. And I admire the packaging team for that to be honest.

What you really think about is not following and bringing back old "setuptools" behaviour is something else that packaging team has already accepted and a number of tools are implementing https://peps.python.org/pep-0723/ - which allows you to define a small subset of pyproject.toml metadata (specifically dependencies) in the single-file scripts. And this is really where yes - any front-end implementing PEP-723 should indeed prepare a venv, install dependencies and run the script that specifies such dependencies.

Anything more complex that really has a bit more complex packaging need - putting more files together, should really define their backend, in order to maintain "build consistency", otherwise you start to be at mercy of tool developers who might change their behaviours at any time and suddenly not only you but anyone else who want to build your package will suddenly have problems with it.

But yes. If uv provides backend to build packages, that you (as project maintainer) specify in your build dependencies, this is perfectly fine - just one more backend to choose among about 10 available today. And if - as maintainer - you will prefer to specify it in your project, you should be free to do so. But as a maintainer, I would never put faith that future versions of specific front-end (including uv) will continue building my package in the same way in future versions.

BTW. Piece of advise - for that very reason, as a maintainer you should do:

[build-system]
requires = ["uv==x.y.z"]
build-backend = "uv.build"

Otherwise you never know which version of uv people have and whether they have a version that is capable of building your package at all (i.e. old version that has no backend).

[potiuk]

Also - again if you have proposal how to improve packaging, there are discourse threads there and PEP could be written, so I also recommend you, if you are strongly convinced that you can come up with a complete and better solution - please start discussion there about new PEP, and propose it, lead to approval and likely help to implement in a number of tools - this is the way how standard in packaging are being developed :)

[notatallshaw]

It's interesting that setuptools remains the "blessed" build backend that pip defaults to use when no backend is specified in the pyproject.toml. Rather than bake setuptools into the stdlib, it silently forces the download and installation of it as an unauthorized third-party build dependency. This also seems to violate the principle of "explicit is better than implicit."

FYI, I beleive this is because pip maintainers are very conservative when it comes to breaking changes, not because it is the intended future of Python packaging.

For example, the old resolver, which can easily install a broken environment, is still available to use even though the new resolver has been available for over 5 years and turned on by default for over 4 years.

[daviewales]

A uv-aware build backend would enable private git packages to depend on other private git packages, and still be installed with pip, pipx, poetry, etc, without needing the package end user to change their tools. See #7069 (comment) for a detailed example. (Poetry packages work this way, as they use the poetry-core build backend.)