please stop pinning to major versions by default / forcing SemVer, especially for Python itself

[NickleDave]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

Hi, I do not find any issues discussing this, so sorry in advance if it's been hashed out before.

My experience with Poetry has been that it's a great tool for development--much cleaner than using a setup.py file, that's for sure. But I don't understand some of the defaults for dependencies.

In particular, I'm confused about why the default is to pin to major versions, for both Python itself and for a project's dependencies.
Especially in the case of Python itself, this can cause problems for new users that switch to Poetry for development.

The default now is to set the Python requirement to >=3.y.z, <4.0, where the important part is the <4.0.
New users will be frustrated when they find that, by using this default, they will suddenly be forcing people to install older versions of things, because the previous setup.py file they used for development probably did not specify "<4.0" for PYTHON_REQUIRES.

This is the situation I am in now.
Unlike you devs, I am not smart enough to understand how the pip or poetry solvers work.
But I am smart enough to know that I can longer install one of my own libraries, according to your solver, because:

  SolverProblemError

  The current project's Python requirement (>=3.6.2) is not compatible with some of the required packages Python requirement:
    - evfuncs requires Python >=3.6,<4.0, so it will not be satisfied for Python >=4.0
  
  Because evfuncs (0.3.1) requires Python >=3.6,<4.0
   and no versions of evfuncs match >0.3.1, evfuncs is forbidden.
  So, because vak depends on evfuncs (>=0.3.1), version solving failed.

  at ~/.poetry/lib/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│ 
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

  • Check your dependencies Python requirement: The Python requirement can be specified via the `python` or `markers` properties
    
    For evfuncs, a possible solution would be to set the `python` property to ">=3.6.2,<4.0"

Pretty sure what's happening to me is what's described here:
https://mobile.twitter.com/HenrySchreiner3/status/1362183452343296004

So poetry is now demanding that I obey semver for Python itself, basically.
i.e, I can "fix" this if I go back to including <4.0 for the package I'm developing now.
But I don't want to, because I don't want anyone depending on my library to be "trapped" the same way I am now.

I'm fine with "opinionated" libraries, but in this case, I'm not aware of any good reason to hold the opinion that my required Python should take semver into consideration. In particular, for Python, there's no reason to assume that 4.0 will be drastically different from 3.x.

Similarly, the default for libraries is to use "^" to pin to a major version, presumably under the assumption that it's somehow safer.
But this can have unintended consequences, as discussed here:
https://hynek.me/articles/semver-will-not-save-you/

But Wait – It Gets Worse!

If you maintain a public package and pin the major version of a dependency of yours, you transitively do this to the applications of your users.

Imagine an application depends on the wonderful urllib3 and your package does too. Now if you pin urllib3 to <2, the user of your package doesn’t have it in their power to ever receive an update from urllib3 again, once urllib3 bumps its major version to 2 and beyond.

They may not even realize how far back they are.

On the other hand, if a new major version of a package surprisingly breaks your package, they can always add a pin themselves (see step 4 above) until you fix your package. But there’s no practical way for them to remove your pin.

Don’t ever pin major versions, unless you know they’re broken.

Some Python packaging tools have adopted npm’s major-version pinning (^) by default, despite the lack of npm’s security features. Make sure to unpin them by hand if possible.

Especially for a lot of data science libraries that stay in "Perpetual ZeroVer" for extended periods of development, this is not a great assumption anyway.
See also this blog post about why semver often doesn't work in practice for Python: https://snarky.ca/why-i-dont-like-semver/

If you really want people to switch to your very helpful tool, please consider some defaults that will not come back to bite people without them knowing it, and are more in keeping with expectations in the community.

In short, why not just always use ">=" by default, for Python and for dependencies?
If the answer is "because semver", I hope you will consider what's in the posts I linked above.

[henryiii]

Using ^<version> is wrong for libraries (and apps should always use the locking system anyway). You should only list known incompatibilities (minimum version supported and known buggy versions as they come up). For Python version; this is as far as you should ever go; the Requires-Python metadata slot is used by pip to scroll back YOUR PACKAGE VERSION to find one that does match, making it only usable for dropping old Python versions. That's it. Never for "future" versions - your most recent version is the most likely version to support Python 4, never an older version. And error message will be completely weird unless you've never released a version without the cap on Requires-Python.

For library requirements, it's a little better, because pip can install older versions. But since Pip 20.3's new solver, it now matters if you over constrain - so it's time for Poetry's default to become >= for poetry add, and for >= to be used in the examples. There are special cases where you know a library is about to release a breaking version, or if you deeply depend on the internals of a library instead of just the public API; but otherwise, you should not add an upper pin.

Let's use an upcoming example. Many libraries depend on click. Usually not even for the functionality they provide as a library. Let's say a few of them (at least two) follow Poetry's recommendation and use ^7. The breaking change in Click 8 is dropping Python 2 and 3.5 support, so if they all disallow version 8, I have to wait until every one of them updates to allow 8 in the app or library I am developing depending on these libraries before I can use it - I can't solve with Click 8. And if any of them use the syntax ^8, I now have a broken solve until all of them support version 8 and there's no fixing it.

If, however, Click 8 does happen to break something unforeseen (as minor / patch versions can break things to; it's just about as common to break something on accident as to break it on purpose in an established library), and no libraries put <8 in; I (the user) can easily fix it by asking for click<8 myself, it's not unsolvable (this recently happened with a minor Jedi release and IPython; a little ugly, but it has a user workaround until a library can pin); and a library can always temporarily add a maximum pin if they know about an inconstancy.

Looking at well designed poetry libraries, like @willmcgugan's rich, I see the ^ syntax being mis/overused, likely due to the defaults and docs here.

[NickleDave]

Thank you, Henry.

I agree that one mistake I made, which is something I should have known as a library developer, was to pin my dependencies to the major version. Although in my defense I did it unthinkingly by following the defaults suggested by poetry. Defaults which I guess are targeted more at application (as opposed to library) developers.

But even if that were not the case, another issue here is the solver being strict and refusing to let me install anything that specifies Python ">=3.x, <4.0" because I myself have only specified Python ">=3.x" (without the "<4.0").

for an example of someone else facing the same issue, see:
nedbat/coveragepy#1020
nedbat/coveragepy#1021

[nedbat]

I'm curious about this:

In particular, for Python, there's no reason to assume that 4.0 will be drastically different from 3.x.

Why do you think this? I think Python versions close to semver. 3.0 introduced breaking changes from 2.x. I think Python would only go to 4.0 if it introduced breaking changes.

[henryiii]

A) The Python devs clearly state that 4.0 (if ever introduced) will not contain breaking changes like 3.0 did, that will not ever happen again. I think the line above is pretty much from the devs. Personally, I think it's likely to happen if the Limited API needs changes; that's tied to "all subsequent 3.x" versions (and might be the one case where "<4" is valid - but as I've mentioned, Requires-Python doesn't handle this correctly - you can't install Python based on your requirements! Also, Limited API can't be used by poetry users (or pretty much anyone, I've only seen one package successfully use it).

B) Every release contains breaking changes. 3.10 has a bunch, like changing the default for annotations to strings, and removing a bunch if Python 2 compatibility stuff. 3.12 will remove distutils. Etc. If you can run Python 3.X without warnings with the -Xd flag, you probably can upgrade a single point version safely. These breaking changes are not usually out-of-the-blue if you follow Python, you might be able to keep updating for years.

C) Even with perfect SerVer, a breaking change may not break your code. For a stable library or Python, the "breakage" likely does not affect light to medium weight users of the code.

[NickleDave]

Thank you @nedbat -- I'm for sure not a core Python dev but what @henryiii matches similar remarks I saw on Twitter

You are totally right that of course there were breaking changes going from 2.0 to 3.0.
So I think some of the remarks were along the lines of: "going from 3.0 to 4.0, if it ever happens, will definitely not be like going from 2.0 to 3.0"

[nedbat]

One thing the core devs haven't said: what would cause them to change the version number to 4.0? My guess is that the public perception of 4.0 would be an assumption of "3.0-like" changes, and the resulting negative PR impact would mean that we aren't going to have a 4.0. But that's just my guess.

[henryiii]

They basically have said - nothing that they know of. I think the Limited API might one thing.

That's besides the point, though - this is the wrong way to use the Requires-Python metadata. If you suddenly realize you don't support 3.10, you should not set <3.10 here. This will not solve by installing an older Python version, it will not produce a pretty error message like "the current version of Python is not supported", instead, it will look for an older release of your software that doesn't list <3.10, then install that, because this feature was designed around dropping Python versions, not eventually supporting them.

For libraries, it's better, since that is part of the standard solver; but still should not be over constrained. If you know Click 8.0 breaks your code, if it's your fault, you can write <8; if it's Click's fault, you can write !=8.0 and file a bug report. But by default you should assume if you use the public, documented API of a package, that's not going to break. New major versions tend to drop Python 2.7 support and things like that, they tend not to completely break "good" users of the packages. Setuptools, for example, randomly breaks everyone on patch releases (for an hour or two) and makes almost no changes on major releases. :P

[henryiii]

PS: Originally, they said things like the string annotations would do it. But that didn't fly, and it was happened in 3.10 instead. Honestly, this is more consistent with historical changes too. New features in the 2.x series like with were added with __future__ in one release, and made standard in the next release (breaking anyone with a with method or function). async/await becoming keywords did the same thing in the 3.x series. And so on.

NumPy also follows this method. Changes have 3 versions of warnings (IIRC) before being made breaking.

[finswimmer]

Hello everyone,

there are different aspects discussed here, that are (more or less) unrelated to each other.

The first one is, that poetry refuses to resolve dependency, if the project hasn't specify an upper boundary for the supported python version, but a dependency of this project has. This is how dependency resolution works. All defined dependencies must be valid for all defined python versions for the project. If a dependency only supports a subset of this, one cannot successfully resolve the dependency. Ignoring the upper boundary of supported python version isn't an option. This would be a wild guess by the resolver, that the package maintainer hasn't carefully think about it. But maybe they had? No one knows. So this behavior by poetry will never change.

The other aspect discussed in this issue, is about the default python version constraint poetry generates. As other stated out, python itself doesn't follow SemVer. But IMO this is not an argument for not defining an upper boundary for the supported python versions. Breaking changes can already occur between minor versions in python. So if a package author haven't test against all available minor versions and don't like to bet, that its package will work in other or future python release, it might be a good idea to restrict the python version constraint to the version they tested. On the other hand, defining the python version constraint in a to small area, makes it hard to resolve dependency if it's a library used in other project. Saying this I believe, the default behavior by poetry, to use the caret operator for python version constraint, is a good compromise. No one knows yet if there ever will be a python version 4 and how it differs from python3. But if there ever will be one, there's a good chance that it will break many things. However my opinion about poetry's default here is not set in stone, but I believe it is a good choice. Furthermore the default is not hide anywhere and it is the responsibility of the package maintainer to review its pyproject.toml and change it if needed.

fin swimmer

PS: I will move this issue to the discussion board.