can poetry be used by library developers / maintainers, e.g. to declare abstract dependencies?

[NickleDave]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

Hi, I do not find any issues discussing this, so sorry in advance if it's been hashed out before.

My experience with Poetry has been that it's a great tool for development--much cleaner than using a setup.py file, that's for sure. But I don't understand some of the defaults for dependencies.

In particular, I'm confused about why the default is to pin to major versions, for both Python itself and for a project's dependencies.
Especially in the case of Python itself, this can cause problems for new users that switch to Poetry for development.

The default now is to set the Python requirement to >=3.y.z, <4.0, where the important part is the <4.0.
New users will be frustrated when they find that, by using this default, they will suddenly be forcing people to install older versions of things, because the previous setup.py file they used for development probably did not specify "<4.0" for PYTHON_REQUIRES.

This is the situation I am in now.
Unlike you devs, I am not smart enough to understand how the pip or poetry solvers work.
But I am smart enough to know that I can longer install one of my own libraries, according to your solver, because:

  SolverProblemError

  The current project's Python requirement (>=3.6.2) is not compatible with some of the required packages Python requirement:
    - evfuncs requires Python >=3.6,<4.0, so it will not be satisfied for Python >=4.0
  
  Because evfuncs (0.3.1) requires Python >=3.6,<4.0
   and no versions of evfuncs match >0.3.1, evfuncs is forbidden.
  So, because vak depends on evfuncs (>=0.3.1), version solving failed.

  at ~/.poetry/lib/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│ 
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

  • Check your dependencies Python requirement: The Python requirement can be specified via the `python` or `markers` properties
    
    For evfuncs, a possible solution would be to set the `python` property to ">=3.6.2,<4.0"

Pretty sure what's happening to me is what's described here:
https://mobile.twitter.com/HenrySchreiner3/status/1362183452343296004

So poetry is now demanding that I obey semver for Python itself, basically.
i.e, I can "fix" this if I go back to including <4.0 for the package I'm developing now.
But I don't want to, because I don't want anyone depending on my library to be "trapped" the same way I am now.

I'm fine with "opinionated" libraries, but in this case, I'm not aware of any good reason to hold the opinion that my required Python should take semver into consideration. In particular, for Python, there's no reason to assume that 4.0 will be drastically different from 3.x.

Similarly, the default for libraries is to use "^" to pin to a major version, presumably under the assumption that it's somehow safer.
But this can have unintended consequences, as discussed here:
https://hynek.me/articles/semver-will-not-save-you/

But Wait – It Gets Worse!

If you maintain a public package and pin the major version of a dependency of yours, you transitively do this to the applications of your users.

Imagine an application depends on the wonderful urllib3 and your package does too. Now if you pin urllib3 to <2, the user of your package doesn’t have it in their power to ever receive an update from urllib3 again, once urllib3 bumps its major version to 2 and beyond.

They may not even realize how far back they are.

On the other hand, if a new major version of a package surprisingly breaks your package, they can always add a pin themselves (see step 4 above) until you fix your package. But there’s no practical way for them to remove your pin.

Don’t ever pin major versions, unless you know they’re broken.

Some Python packaging tools have adopted npm’s major-version pinning (^) by default, despite the lack of npm’s security features. Make sure to unpin them by hand if possible.

Especially for a lot of data science libraries that stay in "Perpetual ZeroVer" for extended periods of development, this is not a great assumption anyway.
See also this blog post about why semver often doesn't work in practice for Python: https://snarky.ca/why-i-dont-like-semver/

If you really want people to switch to your very helpful tool, please consider some defaults that will not come back to bite people without them knowing it, and are more in keeping with expectations in the community.

In short, why not just always use ">=" by default, for Python and for dependencies?
If the answer is "because semver", I hope you will consider what's in the posts I linked above.

[henryiii]

Using ^<version> is wrong for libraries (and apps should always use the locking system anyway). You should only list known incompatibilities (minimum version supported and known buggy versions as they come up). For Python version; this is as far as you should ever go; the Requires-Python metadata slot is used by pip to scroll back YOUR PACKAGE VERSION to find one that does match, making it only usable for dropping old Python versions. That's it. Never for "future" versions - your most recent version is the most likely version to support Python 4, never an older version. And error message will be completely weird unless you've never released a version without the cap on Requires-Python.

For library requirements, it's a little better, because pip can install older versions. But since Pip 20.3's new solver, it now matters if you over constrain - so it's time for Poetry's default to become >= for poetry add, and for >= to be used in the examples. There are special cases where you know a library is about to release a breaking version, or if you deeply depend on the internals of a library instead of just the public API; but otherwise, you should not add an upper pin.

Let's use an upcoming example. Many libraries depend on click. Usually not even for the functionality they provide as a library. Let's say a few of them (at least two) follow Poetry's recommendation and use ^7. The breaking change in Click 8 is dropping Python 2 and 3.5 support, so if they all disallow version 8, I have to wait until every one of them updates to allow 8 in the app or library I am developing depending on these libraries before I can use it - I can't solve with Click 8. And if any of them use the syntax ^8, I now have a broken solve until all of them support version 8 and there's no fixing it.

If, however, Click 8 does happen to break something unforeseen (as minor / patch versions can break things to; it's just about as common to break something on accident as to break it on purpose in an established library), and no libraries put <8 in; I (the user) can easily fix it by asking for click<8 myself, it's not unsolvable (this recently happened with a minor Jedi release and IPython; a little ugly, but it has a user workaround until a library can pin); and a library can always temporarily add a maximum pin if they know about an inconstancy.

Looking at well designed poetry libraries, like @willmcgugan's rich, I see the ^ syntax being mis/overused, likely due to the defaults and docs here.

[NickleDave]

Thank you, Henry.

I agree that one mistake I made, which is something I should have known as a library developer, was to pin my dependencies to the major version. Although in my defense I did it unthinkingly by following the defaults suggested by poetry. Defaults which I guess are targeted more at application (as opposed to library) developers.

But even if that were not the case, another issue here is the solver being strict and refusing to let me install anything that specifies Python ">=3.x, <4.0" because I myself have only specified Python ">=3.x" (without the "<4.0").

for an example of someone else facing the same issue, see:
nedbat/coveragepy#1020
nedbat/coveragepy#1021

[nedbat]

I'm curious about this:

In particular, for Python, there's no reason to assume that 4.0 will be drastically different from 3.x.

Why do you think this? I think Python versions close to semver. 3.0 introduced breaking changes from 2.x. I think Python would only go to 4.0 if it introduced breaking changes.

[charlax]

Those breaking changes could be irrelevant to your library, which might work just fine with this version.

[adam-grant-hendry]

Those breaking changes could be irrelevant to your library, which might work just fine with this version.

Precisely; couldn't agree more. This is Hyrum's Law, which @NickleDave posted from the Hynek article. It's impossible to know a-priori what will be breaking for the end user.

[henryiii]

A) The Python devs clearly state that 4.0 (if ever introduced) will not contain breaking changes like 3.0 did, that will not ever happen again. I think the line above is pretty much from the devs. Personally, I think it's likely to happen if the Limited API needs changes; that's tied to "all subsequent 3.x" versions (and might be the one case where "<4" is valid - but as I've mentioned, Requires-Python doesn't handle this correctly - you can't install Python based on your requirements! Also, Limited API can't be used by poetry users (or pretty much anyone, I've only seen one package successfully use it).

B) Every release contains breaking changes. 3.10 has a bunch, like changing the default for annotations to strings, and removing a bunch if Python 2 compatibility stuff. 3.12 will remove distutils. Etc. If you can run Python 3.X without warnings with the -Xd flag, you probably can upgrade a single point version safely. These breaking changes are not usually out-of-the-blue if you follow Python, you might be able to keep updating for years.

C) Even with perfect SerVer, a breaking change may not break your code. For a stable library or Python, the "breakage" likely does not affect light to medium weight users of the code.

[NickleDave]

Thank you @nedbat -- I'm for sure not a core Python dev but what @henryiii matches similar remarks I saw on Twitter

You are totally right that of course there were breaking changes going from 2.0 to 3.0.
So I think some of the remarks were along the lines of: "going from 3.0 to 4.0, if it ever happens, will definitely not be like going from 2.0 to 3.0"

[nedbat]

One thing the core devs haven't said: what would cause them to change the version number to 4.0? My guess is that the public perception of 4.0 would be an assumption of "3.0-like" changes, and the resulting negative PR impact would mean that we aren't going to have a 4.0. But that's just my guess.

[brettcannon]

My guess is Python 4 will occur either when so much deprecated cruft has accumulated that we need a release to sweep it all out, or we have made a slow transitional change at the C API level and it's time to finally drop the old way of doing C extensions.

[henryiii]

They basically have said - nothing that they know of. I think the Limited API might one thing.

That's besides the point, though - this is the wrong way to use the Requires-Python metadata. If you suddenly realize you don't support 3.10, you should not set <3.10 here. This will not solve by installing an older Python version, it will not produce a pretty error message like "the current version of Python is not supported", instead, it will look for an older release of your software that doesn't list <3.10, then install that, because this feature was designed around dropping Python versions, not eventually supporting them.

For libraries, it's better, since that is part of the standard solver; but still should not be over constrained. If you know Click 8.0 breaks your code, if it's your fault, you can write <8; if it's Click's fault, you can write !=8.0 and file a bug report. But by default you should assume if you use the public, documented API of a package, that's not going to break. New major versions tend to drop Python 2.7 support and things like that, they tend not to completely break "good" users of the packages. Setuptools, for example, randomly breaks everyone on patch releases (for an hour or two) and makes almost no changes on major releases. :P

[henryiii]

PS: Originally, they said things like the string annotations would do it. But that didn't fly, and it was happened in 3.10 instead. Honestly, this is more consistent with historical changes too. New features in the 2.x series like with were added with __future__ in one release, and made standard in the next release (breaking anyone with a with method or function). async/await becoming keywords did the same thing in the 3.x series. And so on.

NumPy also follows this method. Changes have 3 versions of warnings (IIRC) before being made breaking.

[finswimmer]

Hello everyone,

there are different aspects discussed here, that are (more or less) unrelated to each other.

The first one is, that poetry refuses to resolve dependency, if the project hasn't specify an upper boundary for the supported python version, but a dependency of this project has. This is how dependency resolution works. All defined dependencies must be valid for all defined python versions for the project. If a dependency only supports a subset of this, one cannot successfully resolve the dependency. Ignoring the upper boundary of supported python version isn't an option. This would be a wild guess by the resolver, that the package maintainer hasn't carefully think about it. But maybe they had? No one knows. So this behavior by poetry will never change.

The other aspect discussed in this issue, is about the default python version constraint poetry generates. As other stated out, python itself doesn't follow SemVer. But IMO this is not an argument for not defining an upper boundary for the supported python versions. Breaking changes can already occur between minor versions in python. So if a package author haven't test against all available minor versions and don't like to bet, that its package will work in other or future python release, it might be a good idea to restrict the python version constraint to the version they tested. On the other hand, defining the python version constraint in a to small area, makes it hard to resolve dependency if it's a library used in other project. Saying this I believe, the default behavior by poetry, to use the caret operator for python version constraint, is a good compromise. No one knows yet if there ever will be a python version 4 and how it differs from python3. But if there ever will be one, there's a good chance that it will break many things. However my opinion about poetry's default here is not set in stone, but I believe it is a good choice. Furthermore the default is not hide anywhere and it is the responsibility of the package maintainer to review its pyproject.toml and change it if needed.

fin swimmer

PS: I will move this issue to the discussion board.

[finswimmer]

Another aspect discussed here, I missed in my first reply is, using the caret requirement for dependencies by default. IMO this is a very good choice and should not be changed. SemVer is widely used, even if it has its problems in the real world. It's the best predictor we have, whether a version bump may change our library or not.

[NickleDave]

Hi @finswimmer thank you for taking the time to respond.

You are right there are different aspects to the issue I raised.
I think the main question is: can library developers/maintainers use poetry to more easily develop their libraries.
At this point, it seems like the answer is no, because of reasons I'll outline below, and that poetry is mainly targeted to developers of apps.

Now that you've moved this to Discussions (thank you), let me try to state everything more clearly.

Many of the points you make about dependency resolution and SemVer are very valid for app developers, who primarily want their app to be stable in production. Although reasonable people can disagree about the benefits of SemVer, as outlined in those posts from Hynek Schlawak and Brett Cannon.

But the same points do not make sense for library developers.

As a library developer, I want to define my "abstract dependencies" in the pyproject.toml, to get away from all the problems with declaring them in a setup.py file.

I thought that poetry would help me with that, as a naive and inexperienced developer, but it seems like I was wrong.

Just to be extra clear, my current understanding is that a library developer should not pin their dependencies, including Python, for several reasons. For example, here on this Python Packaging Authority page it states:

It is not considered best practice to use install_requires to pin dependencies to specific versions, or to specify sub-dependencies (i.e. dependencies of your dependencies). This is overly-restrictive, and prevents the user from gaining the benefit of dependency upgrades.

https://packaging.python.org/discussions/install-requires-vs-requirements/#install-requires

I see that poetry does pin its own dependencies in the pyproject.toml file. I'm afraid that if you are building from that file, then you are in fact preventing your users from gaining the benefit of dependency upgrades. I think this is the problem @joelb123 is running into here:
#2731
(although it sounds like some of their problem might be resolved by using virtual environments?)

It's also the case that a library developer never wants to upper-bound the required version of Python, I think, as @henryiii described above.

For Python version; this is as far as you should ever go; the Requires-Python metadata slot is used by pip to scroll back YOUR PACKAGE VERSION to find one that does match, making it only usable for dropping old Python versions. That's it. Never for "future" versions - your most recent version is the most likely version to support Python 4, never an older version.

I don't understand all the nuances of that, but for any library developer reading this: please do be aware of issues you might run into if you switch to poetry and accept the default pinning to a major version of Python (see related discussion on this issue with @nnadeau and @nedbat nedbat/coveragepy#1020)

So:
I want to use the pyproject.toml to declare install_requires, and I thought that I could do that using poetry.
Put another way, I want to use pyproject.toml as my .gemspec file, as @k0nserv describes here
pypa/pipfile#27 (comment)

That's also exactly what @bryevdv wants to do here:
#3738

It seems like this is still not a possibility, and the current standard, which is not explicitly written down anywhere, is:

• for library developers to continue to use setup.py (or possibly setup.cfg, see below)
• while app developers use pyproject.toml for their dependencies
• and poetry.lock for the exact versions used in deployment.
  As @Screwtapello states here:
  The difference between setup.py (pyproject.toml) and requirements.txt (Pipfile) pypa/pipfile#27 (comment)

I think one reason why this gets confusing is that poetry forces users to declare metadata in the tool.poetry section, instead of getting that metadata from a "neutral" section of the pyproject.toml file. This is the "vendor lock-in" that @Kwpolska refers to: https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/#poetry-better-but-still-not-convincing
(cześć Chris, sorry for tagging you in this random issue)

@taion says something similar here where he says
pypa/pipfile#98 (comment)

In an ideal world, this would be a non-issue if we had a setuptools entry in pyproject.toml, but we don't have that right now.

To be clear, I'm not sure what that section would be or how it should be named, and I think the proposal described in that issue for pipenv is probably not a good long-term solution.
But the description @Taoin gives of the issues with library v app development is one of the dlearest I have read so far.

As a final point related to this, it is possible to declare install_requires in a static file that is not setup.py; that's the setup.cfg which in the end is used by setuptools ( @bryevdev maybe this an alternative solution for you?)
https://python-packaging-tutorial.readthedocs.io/en/latest/setup_py.html#setup-cfg
But of course this is not the pyproject.toml file, and it would be nice if library devs could just put everything in one file. And I think we can all agree that ini files are kind of gross (now I'm going to get angry replies from the original developer of the ini file format 😆 ).

Hope that summary helps anyone else considering similar questions, and happy to edit + hear feedback

[joelb123]

I find this library vs app distinction to be rather precious. When I write code, I don't know whether anyone will choose to re-use my code as a library sometime in the future, nor do I have control of it.

What I gather is that if I don't want a change in major version number by a dependency to break my automated upgrade and testing processes or break future installations when the pinned dependency has gone obsolete, then I need to specify mydependency = "*". However, there isn't a way to do that except by manually editing the pyproject.toml. What I hate about that is that I haven't tested against lower versions of dependencies than latest at the time I added them, so saying that any version will work is a lie.

[henryiii]

You should always test against the lowest version of your supported dependencies (usually I am too lazy and instead base it on what should work based on reading changelogs, but this is what you should do). Currently, for pypa/build, we have a constraint file and an explicit test for it. It's great if you have a dependency resolver feature for that - there's an open issue requesting one for pip - see pypa/pip#8085. Might be a good idea for poetry if there's not already an option for that.

When I write code, I don't know whether anyone will choose to re-use my code as a library sometime in the future

Do you have a public API, and docs describing how to use it as a library? If so, it's a library. Do you have docs describing how to use it as an app? It's an app. Yes, people may not respect this - this is the reason pip 10 was so controversial (CentOS 8 still ships pip 9). They considered themselves an app, so they moved everything in import pip to import pip._internal in pip 10, and (intentionally) broke tons of things, including the pip script from older versions of pip (which were often not user upgradable when coming from distributions).

But, in general, either you design something as an app, or a library, or both, but you know what you design for. If you are targeting library usage, you should never add maximum version pins unless you a) heavily use an API, or b) know something is changing that will break you. 90% of stable libraries will not break for you in their next major release. But sometimes a library has to pin a high minimum version, like importlib_resources>=3.6. If someone set importlib_resources=^2 for example, this will be unsolvable!!! And that's a backport from the standard library, upgrading it will not break your usage.

[Agalin]

As someone who works with both Python and JavaScript - I don't think that python world can work with semver dependencies for libraries (even though I'm personally check major version changes more closely) considering current import logic.

Currently packages often don't use SemVer release style. In theory poetry gaining popularity could affect that but it doesn't really matter. Why? How can it work for JS but not for Python that is just as dynamic? Because we're missing support for transitive dependencies. In JS world if dependency A wants Click ^8 and B wants Click ^9 instead of failing (as it is in Python) it would install both versions and provide those accordingly to A and B - and our main app can even use another version.

There is also one more thing supported in JS but not in Python (this one could in theory be supported by Poetry but it's more like a hack) - overriding dependencies of dependencies. If you know that A should work with Click 9 you can force it.

So IMO either poetry should support defining a default dependency limitation for add (so you can set it to ^ or >= as you wish), document that issue (or at least link here) and... maybe pass it to authors of PEP582 so transient dependencies could be discussed?

[henryiii]

A Python core developer that works on packaging (@brettcannon) wrote:

Libraries/packages should be setting a floor, and if necessary excluding known buggy versions, but otherwise don't cap the maximum version as you can't predict future compatibility

I still believe it is very bad for Poetry to both recommend and default to capping all versions, and then to enforce child projects to also cap versions if they use something that caps versions. (To be clear, I'm not advocating that it not enforce that, just that it not recommend capping / default to capping in the first place). Click's update to version 8 recently highlighted this problem - it does not scale.

And it's not really because SemVer is "wrong", it's because bumping a major version is not a promise that you will break every user of your library. It's an indicator that you think you might break someone. It is very unlikely that a normal user really cares about Click 7 vs. 8, or Pytest 5 vs. 6, etc. Dropping Python 2 is a common reason to bump major version. Changing internal API is another.

@willmcgugan's Rich is an example of a very popular library that caps versions because Poetry recommended it. I'm afraid to depend on Rich for an important project because of it - I don't want to get version conflicts when something updates. Which is a shame because I love color.

[joelb123]

I agree with Henry's point of view, default capping is a bad idea. To be specific, the worst thing about defaulting to a cap is that there is no way to distinguish between affirmative knowledge that a package fails above a certain version (I'm thinking pydantic and python 3.10) and a cap based on untested presumptions.

[henryiii]

https://iscinumpy.dev/post/bound-version-constraints/ and https://iscinumpy.dev/post/poetry-versions/ are followups to this discussion, FYI.

[henryiii]

Both links work for me. What is it showing?

[jimmywan]

"Could not resolve host: iscinumpy.dev"

[henryiii]

That's weird. It's just a google domain name. The new domain name is sort of new (few weeks old), so maybe you have an old DNS server? Anyway, https://iscinumpy.gitlab.io/post/bound-version-constraints/ and https://iscinumpy.gitlab.io/post/poetry-versions/ should work.

[brettcannon]

I can hit the links without issue.

[jimmywan]

Ahh, I see it now. Looks like it's being blocked by Quad9, my upstream DNS provider.