Get rid of hatchling and return back to setuptools

[KOLANICH]

In PR #338 setuptools was replaced by hatchling by its maintainer, who obviously has conflict of interest between promoting his software and doing the best for this project as a contributor. The PR was sent 2022-03-27, setuptools 61.2.0 with usable support of PEP 621 was released the same day (setuptools has started supporting PEP 621 in 61.0.0 released on 2022-03-24, but there were issues, so I usually set >=61.2 in my projects).

We need:

1. a lightweight solution
2. requiring no additional dependencies

setuptools is the de-facto standard and so should be already installed. I think we should replace hatchling with setuptools. If setuptools seems to be too heavy, the next alternative is flit. All of them support PEP 621, so the changes in the pyproject.toml should be minimal.

And yet another opinion: PEP 517 was a big mistake because instead of one single build system (setuptools) we now have to install a whole zoo of build systems (flit_core, poetry_core has long become a mandatory minimum with its own very loyal adepts (worth noting: flit really works much faster than setuptools, so using flit may be justified in some cases), and now hatchling and maturin (an extremily heavy build system written in Rust, even though setuptools-rust plugin exists for this purpose) seems to be added to it) doing the same things but created and promoted out of "not-invented-here" syndrome.

[wiggin15]

Well, this is of course debatable, but IMHO PEP 517 was not a mistake, because standards are better than a single implementation that encapsulates all the details of a specific operation. There is nothing wrong with multiple implementations of the same thing as long as everyone adheres to some API / standard. It's like there are multiple clients and servers that exist for HTTP but they are all "speaking" the same protocol. Or like Python has multiple implementations but they are all implementing the same language. Having only setuptools was a problem that manifested in multiple ways - if you remember the days of "easy_install" and "egg"s. Every time a standard needed to be set.
I'm hoping to avoid an "internet argument" about this though - PEP 517 is accepted so it is a done deal.

I was skeptical about hatchling at first myself, but what convinced me it is better than setuptools was that seuptools is missing support for some features without a setup.py file. For example, you can't run pip install -e . to install the package in edit mode when the build backend is setuptools - it requires a dummy setup.py file for some reason. hatchling does not.
Another thing that convinced me is that it is recommended by PyPA (flit is also on this list).

Regarding "requiring no additional dependencies" - this is not a runtime dependency but a build time dependency, so it is not a real problem. Note that we aren't using the entire hatch ecosystem, just its build backend - hatchling.

I am not familiar with flit, so if you can explain the benefits of using it over hatchling then I'm open for discussing it.
It doesn't appear to be more lightweight than hatchling at least in terms of download size from PyPI and dependencies... The source distribution for hatchling is 47.3 kb and requires no extra dependencies. flit is 136.6 kb and has 5 direct dependencies that need to be installed if we decide to use it.

[KOLANICH]

There is nothing wrong with multiple implementations of the same thing as long as everyone adheres to some API / standard.

Sure, but PEP 517 is used to require an exact implementation, not any implementation supporrting the standard. PEP 621 even have not been created when PEP 517 has been issued. And even now PEP 621 doesn't cover several important aspects, so each tool uses an own section in pyproject.toml to encode them.

IMHO the more sound approach would have been to develop a standard first and only then encourage third-party impls, and do it the way that a user can use any impl following the standard.

if you remember the days of "easy_install" and "egg"s.

I have never used them in my projects. Well, eggs were used somehow, but not in form of packages, but I remember that some URIs used to get packages directly from GitHub contained the part with the name egg.

For example, you can't run pip install -e . to install the package in edit mode when the build backend is setuptools - it requires a dummy setup.py file for some reason. hatchling does not.

1. It was the bug in pip, not in setuptools.
2. I have fixed that bug long ago. - 2021-02-01, landed into release 2021-04-24.

this is not a runtime dependency but a build time dependency, so it is not a real problem.

For me it is a problem. My CI pipelines install most of the deps from sources. And fetching anything from pypi is disabled - if a dependency is added, the build breaks, and I add it manually.

But build systems are different beasts and have to be handled separately. I always install flit_core because it is a dependency of some essential packages like build, but for other build systems ... I just managed to stay away of packages using them untill now. Now I have to redo my CI scripts in order to support things like hatchling (we surely don't want to install them on every build if they are not needed).

am not familiar with flit, so if you can explain the benefits of using it over hatchling then I'm open for discussing it.

I have not used hatchling untill today, but there is at least one advantage of flit_core over hatchling: it is already installed anyway. We don't need 100500 mandatory build systems, we need one good build system.

It doesn't appear to be more lightweight than hatchling at least in terms of download size from PyPI and dependencies... The source distribution for hatchling is 47.3 kb and requires no extra dependencies. flit is 136.6 kb and has 5 direct dependencies that need to be installed if we decide to use it.

flit is the elder brother of flit_core, the same way hatch is an elder brother of hatchling and poetry is an elder brother of poetry-core. It is incorrect to compare flit to hatchling. flit_core 3.7.1 takes 60.2k in the form of a wheel, hatchling 1.6.0 takes 62.5k, so in terms of size they are prety identical and both work much faster than setuptools, IDK the origins of the slowdown in setuptools, I can hypothesize it is because setuptools has plugins, which are set up through entry points (parsing entry points causes python to walk file system and parse files) and are always invoked (I know it as an author of one of plugins for setuptools).

[wiggin15]

It was the bug in pip, not in setuptools.
I have fixed that bug long ago. - 2021-02-01, landed into release 2021-04-24.

setuptools still isn't working for me when trying to install in edit mode. With latest pip (22.2.2) and setuptools (63.4.1). Running pip install -e . results in:

colorama has a 'pyproject.toml' and its build backend is missing the 'build_editable' hook. Since it does not have a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable mode. Consider using a build backend that supports PEP 660.

It is also documented here.

I have not used hatchling untill today, but there is at least one advantage of flit_core over hatchling: it is already installed anyway.

Can you pleas explain why / where it is installed anyway? Does it come installed by default in Linux distributions, or comes with Python installations? As far as I know I don't have it installed on my development machine currently.

We don't need 100500 mandatory build systems, we need one good build system.

colorama is only using one build system, and since I am not thoroughly familiar with all the options, I don't have enough information to decide what makes a build system "good" or better than an alternative. Switching hatchling with flit seems to me like just replacing one single system that's working with another one that might do the same.

flit is the elder brother of flit_core, the same way hatch is an elder brother of hatchling

I see. So you're saying the build backend should be flit_core? Can you give an example of how to configure pyproject.toml to use flit?

[ofek]

To enumerate, Hatchling is everywhere:

• Debian
• Fedora
• Arch Linux
• conda-forge
• Nixpkgs
• Alpine Linux
• FreeBSD
• Gentoo Linux
• MacPorts
• OpenEmbedded
• Spack

Please tell me where it isn't and I will resolve it promptly.

[ofek]

There is nothing wrong with multiple implementations of the same thing as long as everyone adheres to some API / standard.

Sure, but PEP 517 is used to require an exact implementation, not any implementation supporrting the standard.

No, @wiggin15 is correct here: any implementation supporting the standard is equally valid.

[KOLANICH]

setuptools still isn't working for me when trying to install in edit mode. With latest pip (22.2.2) and setuptools (63.4.1).

It seems they have broken it again:

1. I personally have installed the version of setuptools supporting PEP660: Merge PEP 660 implementation into main pypa/setuptools#3488 , but for me editable install has worked (I mean it started and did something, but whole setup.py develop got broken somewhen for Debian-based distros (installs stuff by wrong paths because Debian devs have not placed the needed config files to the needed place), but PEP660-based mechanism works, so I have used that branch on my local machine as a workaround)
2. if I remember right there should be an option for PIP to ignore PEP660 and use legacy mechanism when doing editable installs.

I'm sorry, I have no time right now to investigate why that has got broken.

Can you pleas explain why / where it is installed anyway?

Some essential packages like pep517, which is used by build (the CLI tool required to build pep517 and flit_core themselves), depend on flit_core. If one builds everything from source (this implies disabling build isolation and fetching from pypi), he has to mess with PYTHONPATH to build and install everything.

Does it come installed by default in Linux distributions, or comes with Python installations?

AFAIK no, but build "installs" (into a temporary directory on each package installation, if build isolation (a yet another useless wasteful cargo cult) is enabled (it is enabled by default, -x flag to disable it when invoking build explicitly, and pip also has a pref)) stuff mentioned in build-system.requires automatically from pypi, and pretty a lot of packages build process depends on flit_core now.

Switching hatchling with flit seems to me like just replacing one single system that's working with another one that might do the same.

Yes, it's true. But the benefit is: less packages to be installed. In fact I'd have preferred setuptools over flit_core because of exactly the same reason: using more popular packages instead of less popular ones decreases the chances of additional installation of less popular ones. There is nothing wrong with hatchling except that it does the same thing as the other packages that are more popular. Any additional dependency is bloat and security risk, and installing it in CI is an additional delay, so minimizing mandatory dependencies should always be a goal.

I see. So you're saying the build backend should be flit_core? Can you give an example of how to configure pyproject.toml to use flit?

https://github.com/pypa/pep517/blob/main/pyproject.toml

[ofek]

Can you explain the issue you or your company is experiencing adding Hatchling to builds?

[KOLANICH]

The issue is: I have to install yet another python package. It is a burden in multiple ways.

1. It is an additional security risk.
2. It is an additional breakage risk.
3. I have to take care of installing hatchling in my CI pipelines. Currently the code within my CI pipelines are pretty dumb bash scripts, to support building the packages depending on anything other than setuptools and flit_core I have to write a tool. I have already started writing it. But while I'm writing it, my CI pipelines are broken.
4. It is a dependency on your package controlled by you. It gives you power over dependent packages, and so over their users, including the packages dependent on them, and so on. One day you will decide "I hate that CPython version and drop its support", that will cause the most of packages depending on yours to do the same. I don't want you or anyone else to decide which versions of python I should support support in my packages. I don't want you or anyone else to decide that I must maintain a fork of everything. I'd also prefer to avoid implementing breakable technical workarounds to organizational problems in my software. I currently see no value in using your hatchling tool instead of flit_core. I understand that you have done work, that works pretty well, and that you want fame and influence as a compensation, but I think that that niche was already taken and that fragmenting the ecosystem does more harm than the benefit existence of your tool gives. I think that if you see that you can improve the existing tools your first try should have been improving them, not making a new tool and promoting it by making cold PRs into repos. Because you will have hard time to explain people why they need your tool instead of already established and more popular solutions. Network effects are present in software ecosystems too, the winner gets the everything, and the current winner is setuptools, and without will and involvement of PSF into making the ecosystem more fragmented such tools like flit and poetry wouldn't have suceedeed (the fact they were capable to take at least some share of setuptools's share is a big success for them). I completely understand your drive to promote your tool this way, because I do the same with mine, but my tools don't aim to replace setuptools, instead they augment it (and poetry-core, which is also capable to call setuptools plugins).

I see nothing wrong in your tool (I have not analysed it). What is wrong, is that it is a yet another tool doing the same thing setuptools, flit_core, and poetry-core do. I don't need it. I already have setuptools installed. I already have flit_core installed. I don't want to depend on hatchling and 100500 of other tools of other people, each of which needs fame, influence and money, can use their influence to drive the ecosystem to what they want, can be compromised by APTs or any other cybercriminals to utilise their influence to spread malware, can be bribed to sell their influence and reputation for money to spying companies (for example there have been multiple cases of credentials to upload packages and browser extensiins being sold), or can just go rogue like authors of left-pad, colors or node-ipc had done.

I don't want your hatchling. I see nothing wrong with it by itself (I have not analysed it), but it is an additional dependency, I don't need it.

[ofek]

It is an additional security risk.

Less so than setuptools as the codebase is an order of magnitude smaller and thus more auditable.

It is an additional breakage risk.

So is any non-build dependency. Should projects just never add new deps in your ideal packaging ecosystem?

to support building the packages depending on anything other than setuptools and flit_core I have to write a tool. I have already started writing it. But while I'm writing it, my CI pipelines are broken.

This sounds like a problem specific to you. Many companies have and are switching without issue.

It is a dependency on your package controlled by you. [...] One day you will decide "I hate that CPython version and drop its support", that will cause the most of packages depending on yours to do the same. I don't want you or anyone else to decide which versions of python I should support support in my packages.

That is the nature of dependencies and has nothing to do with this discussion. FYI Hatch will drop support for Python versions on the same day they reach EOL.

I currently see no value in using your hatchling tool instead of flit_core. [..] yet another tool doing the same thing setuptools, flit_core, and poetry-core do.

That indicates to me that you haven't made any attempt at all to learn the benefits/differences and why people are switching.

I understand that you have done work, that works pretty well, and that you want fame and influence as a compensation, but I think that that niche was already taken and that fragmenting the ecosystem does more harm than the benefit existence of your tool gives.

I want to improve Python packaging. Would you say Chrome improved the web browser space at all, or only fragmented it to everyone's detriment?

I think that if you see that you can improve the existing tools your first try should have been improving them

Sometimes that's untenable, please read https://peps.python.org/pep-0517/#abstract

I don't need it.

Great, but others do.

[KOLANICH]

Less so than setuptools as the codebase is an order of magnitude smaller and thus more auditable.

That's true. But setuptools are already here and we have to use them, and it is a de-facto standard package tool. We cannot get rid of setuptools. But we can improve it.

So is any non-build dependency. Should projects just never add new deps in your ideal packaging ecosystem?

That is the nature of dependencies and has nothing to do with this discussion.

Every use of every dependency should be justified. Benefits should outweight drawbacks in order for it to be justified. I don't see that depending on your tool is justified: we can do the same (build a wheel from a cloned git repo) using the packages that though are not part of the standard library of CPython, but that can still be considered a part of the language and present on every machine where building of python packages is done.

That indicates to me that you haven't made any attempt at all to learn the benefits/differences and why people are switching.

I don't care, 2BH. I only need it to build wheels. Every additional benefits are too minor to justify depending on it. Yes, your tool is faster than setuptools. But it just means that setuptools should be improved to make it faster.

This sounds like a problem specific to you. Many companies have and are switching without issue.

Yes, it's true.

FYI Hatch will drop support for Python versions on the same day they reach EOL.

It is a problem. CPython drops python versions too fast. I still use 3.4 on some Windows XP machines (setuptools has dropped 3.4, I'm thinking about creating a tool automatically backporting packages to 3.4 because usually old versions of python are dropped just to get fancy-looking syntax sugar (not really needed) and some stdlib packages (lot of which work flawlessly if just copied from newer python versions into the stdlib dir of older ones) ). I still use Windows 7 on some machines, which has also been droped by CPython just using a claim "It's EOL" as an excuse. No, "its creators have declared it EOL" is not an excuse to drop support of anything. Creators of every software always have a conflict of interests and when they declare anything as EOL, it just means they prioritize their interests over interests of users of their software. When someone says "we EOL something because someone else has EOLed that thing", the real meaning behind it is "I also have a conflict interests and I just wanna drop support of as much things as possible, here I got an excuse and I'll use it".

Sometimes that's untenable, please read https://peps.python.org/pep-0517/#abstract

I have read that. I consider

(c) it’s very difficult to use anything else, because distutils/setuptools provide the standard interface for installing packages expected by both users and installation tools like pip.

a feature, not a bug, because it forced everyone to use setuptools and we didn't have the issue with the mess when everyone inventing his own wheel-building system just out of NIH-syndrome instead of just improving setuptools. Now we have a mess. Everyone can demand his own build system to be used. And his package will be installed from sdist. If the possibility to demand a particular build system hadn't been provided to packages authors, but to users via a CLI and config options, and the standard like PEP 621, but covering all the aspects (I mean that even PEP 621-compliant backends need tool-specific sections in pyproject.toml because not everything is standardized) had been issued before PEP 527, it would have been much better.

[ofek]

Can you maybe express your frustration on a different tracker? I don't want to bother the maintainers here. Feel free to email me too!

[wiggin15]

I can relate with the frustration of having to fix broken pipelines because of changes in dependencies, and having to support different packages with their unique sets of dependencies. However, if I understand the issue correctly, this is an edge case that is specific to this pipeline. Another user might have their pipeline configured to install hatchling for something else, but not flit-core, and then we'll have the same problem in the opposite direction.
setuptools is currently out of the question because they lack support for standards like PEP 660. We can live without editable installs, but from what I can tell they are debating over how to implement things and are preparing a complex solution - all of that is a red flag as far as I'm concerned, so I prefer to stay back. If hatchling ever proves problematic, we can switch - that is why it is good to have options.
I think the issue here is being exaggerated - colorama added just one dependency, a build time dependency - it is a legitimate thing to do that does not affect the majority of our users.

It looks like this is about an opinion on PEP 517 and its different implementations more than it is about colorama. Right now I'm not convinced there is a need for a change in colorama so I'm closing this.
I also don't like the tone of this conversation and the exaggeration, so I'm locking this issue. If there is ever a legitimate concern please open a new issue.