Cannot install Apache Airflow due to ambiguous constraint

[r-richmond]

• Poetry version: Poetry (version 1.2.2)
• Python version: Python: 3.10.8
• OS version and name: MacOS 13.0.1
• pyproject.toml:

[tool.poetry.dependencies]
python = "^3.10"
apache-airflow-providers-google = { version = "==8.5.0rc1", allow-prereleases = true }

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
  • In version-constraints prefix-matching (with .*) is not respected after a pre-release version #6774 looks related but is a different error as that is installing incorrect packages
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

I am unable to run poetry lock with the above dependency.

Expected

Poetry locks correctly using apache-airflow-providers-common-sql==1.3.0rc1

Actual

Poetry returns the following error

poetry lock

Updating dependencies
Resolving dependencies... (1.5s)

Because apache-airflow-providers-google (8.5.0rc1) depends on apache-airflow-providers-common-sql (>=1.3.0) which doesn't match any versions, apache-airflow-providers-google is forbidden.
So, because poetry depends on apache-airflow-providers-google (==8.5.0rc1), version solving failed.

It looks like poetry has the following bug

Poetry is not supported to install airflow, and it has apparently a bug that it cannot properly handle the vaild >= .* requirement.

Note: the reproducibility of this example will stop once airflow publishes the final release of apache-airflow-providers-common-sql==1.3.0

Verifying that pypi has the expected requirement

API response

import json

import requests

package_name = 'apache-airflow-providers-google/8.5.0rc1'
url = 'https://pypi.python.org/pypi/' + str(package_name) + '/json'
data = requests.get(url).json()

print(data['info']['requires_dist'])

for apache-airflow-provider-common-sql returns the following

apache-airflow-providers-common-sql (>=1.3.0.*)

Further Notes

pipenv & pip run without issue

Pipenv

Pipfile

[packages]
apache-airflow-providers-google = { version = "==8.5.0rc1", pre="true" }

CLI

mkdir test-pipenv && cd test-pipenv
pip install pipenv
pipenv lock
nano pipfile

pipenv lock                                                                                                                                                                                                                   
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Locking [dev-packages] dependencies...
Updated Pipfile.lock (43302be42e771518e472473549c7ab11902fd6f2e7eed04c64ffcd2d5f9be4e8)!

Locked Packages

...
        "apache-airflow-providers-common-sql": {
            "hashes": [
                "sha256:9308d3ac4d1f888f1a1f91c18cfa493291e333635cf0a0f93565581a83b1f097",
                "sha256:f5eb101f15a3535f879509873992d7194ef3e2a48316c984a0fc2a3a3af9001e"
            ],
            "markers": "python_version ~= '3.7'",
            "version": "==1.3.0rc1"
        },
        "apache-airflow-providers-google": {
            "hashes": [
                "sha256:a6d94a647c4dca9b2b75b4bf98c20c3809eb4ff8ccf4e7219f19df8fec3ea0de",
                "sha256:db9105714472b9cf344b55a72172dd87d85b7de5c8c5755bd86007c23d3ec3cf"
            ],
            "index": "pypi",
            "version": "==8.5.0rc1"
        },
...

[potiuk]

Just to note here, that until constraint support is implemented in Poetry, even fixing this problem will not change our policy - currently only pip is supported to install airflow in reproducible way. The current state of Airlfow installation from PyPI is described here:

https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html

While there are some successes with using other tools like poetry or pip-tools, they do not share the same workflow as pip - especially when it comes to constraint vs. requirements management. Installing via Poetry or pip-tools is not currently supported. If you wish to install airflow using those tools you should use the constraints and convert them to appropriate format and workflow that your tool requires.

[r-richmond]

Just to note here, that until constraint support is implemented in Poetry, even fixing this problem will not change our policy - currently only pip is supported to install airflow in reproducible way.

That's fine. I don't expect airflow to support poetry and am not pursuing that goal.

I linked the original issue just for context but primarily made this issue since the error I ran into seems like a poetry bug and I assume users/maintainers of poetry would like it fixed.

[dimbleby]

not a bug, because the requirement is >=1.3.0 and the highest available version is 1.3.0rc1 - which is strictly less than 1.3.0

(possibly also duplicate #4405, though I've not read the whole of that issue)

[potiuk]

Nope. the requirement is >=1.3.0.* actually. And pip recognizes it.

[potiuk]

You can see the package with this requirement here: https://pypi.org/project/apache-airflow-providers-google/8.5.0rc1/

[potiuk]

Relevant part of METADATA in .whl file:

Requires-Python: ~=3.7
Description-Content-Type: text/x-rst
License-File: LICENSE
License-File: NOTICE
Requires-Dist: PyOpenSSL
Requires-Dist: apache-airflow-providers-common-sql (>=1.3.0.*)

[r-richmond]

I'm at risk of being out of my depth here but the following statement

not a bug, because the requirement is >=1.3.0 and the highest available version is 1.3.0rc1 - which is strictly less than 1.3.0

Appears to be at odds with the spec defined in pep 440
Specifically the last portion of this sentence

Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, unless they are already present on the system, explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release.

Further Details

From the following section (Bullet 2)

• By default, dependency resolution tools SHOULD:

  • accept already installed pre-releases for all version specifiers
  • accept remotely available pre-releases for version specifiers where there is no final or post release that satisfies the version specifier
  • exclude all other pre-releases from consideration

From the spec on Request Options

• Dependency resolution tools SHOULD also allow users to request the following alternative behaviours:

  • accepting pre-releases for all version specifiers

I believe this is what the intent of poetry's allow-prereleases is as well. so even if poetry wanted to break the implied behavior above shouldn't my usage of that flag in pyproject.toml have allowed the pre-release.

[dimbleby]

@potiuk I don't see any justification for >=1.3.0.* in https://peps.python.org/pep-0440/#inclusive-ordered-comparison.

==1.3.0.* would be allowed per the previous section (and quite possibly mis-handled by poetry) but the >= operator requires an actual version identifier - which 1.3.0.* is not.

@r-richmond that still requires that "the available version satisfies the version specifier". poetry has done its best with the malformed version specifier, interpreting it as >=1.3.0, but no available version does satisfy that.

[potiuk]

I think, then it is the question to settle between Poetry and Pip. It works the way it works for pip. It does not for poetry . >=1.3.0 will not install rc - neither via pip nor via poetry. But >=1.3.0.* will work for pip and won't work for poetry.

If there is other way I can make installation of one RC to be dependent on another - I am happy to apply it.

[potiuk]

cc: @uranusjr - maybe you can comment on this ?

[dimbleby]

I think ==1.3.0.* achieves what you want per PEP 440 (though I suspect that poetry will make a mess of that). Also >=1.3.0rc1 should work (and I think poetry will get that right).

So far as I can tell PEP 440 simply doesn't allow >=1.3.0.* and so it's presumably more or less arbitrary whether pip or poetry happens to do what you want

[r-richmond]

r-richmond that still requires that "the available version satisfies the version specifier". poetry has done its best with the malformed version specifier, interpreting it as >=1.3.0, but no available version does satisfy that.

When reading about the differences between Exclusive ordered comparison & Inclusive ordered comparison
it makes the following statement in the exclusive section

The exclusive ordered comparisons > and < are similar to the inclusive ordered comparisons in that they rely on the relative position of the candidate version and the specified version given the consistent ordering defined by the standard Version scheme. However, they specifically exclude pre-releases, post-releases, and local versions of the specified version.

Given that exclusive ordered comparisons's specific exclusion of pre-releases is listed as a difference between it and inclusive-ordered-comparison that implies that inclusive-ordered-comparison does implicitly include pre-release versions.

This is in agreement with the earlier statement in pep440 By default, dependency resolution tools SHOULD: 2) accept remotely available pre-releases for version specifiers where there is no final or post release that satisfies the version specifier.

Further if you disagree with the spec here due to ambiguity (of which there definitely is) is there a reason that passing the allow-prereleases option wouldn't work. As pep440 also states

Dependency resolution tools SHOULD also allow users to request the following alternative behaviours:

• accepting pre-releases for all version specifiers

[dimbleby]

@r-richmond excluding and including pre-releases is a separate question than whether or not the version satisfies the constraint

eg >=1.3.0 is satisfied by 1.3.1rc1 but not by 1.3.0rc1. Only once that standard is met can you ask whether or not the 1.3.1rc1 pre-release is allowed as a pre-release.

[r-richmond]

excluding and including pre-releases is a separate question than whether or not the version satisfies the constraint

eg >=1.3.0 is satisfied by 1.3.1rc1 but not by 1.3.0rc1. Only once that standard is met can you ask whether or not the 1.3.1rc pre-release is allowed as a pre-release.

Ah I see, that is not how I read the spec but I can follow that logic.

Unfortunately I don't see any examples listed in pep440 that explicitly disambiguate this situation :S Is there another PEP worth referencing here?

[dimbleby]

Just noting that indeed poetry does make a mess of prefix matching on == constraints: this code

#!/usr/bin/env python3

from poetry.core.constraints.version.version import Version
from poetry.core.constraints.version import parse_constraint

constraint = parse_constraint("==1.1.*")
version = Version.parse("1.1a1")
allowed = constraint.allows(version)
print(f"{allowed=}")

says allowed=False which certainly disagrees with one of the examples in PEP440.

So while - at least according to my reading - the problem in this particular issue is the airflow constraint rather than the poetry handling of it, we're certainly very adjacent to a poetry bug

[potiuk]

I do not think this is "airflow constraint" problem. I think simply currently there is no very well and non-ambigious way to say "depend on any pre-release version of this package but minimum is x.y.z.<whatever pre-release>".

And this is a valid use case if you want to release two dependent packages and one of them depend on each other, and you want people to test them at the same time.

The >=x.y.z.* seems to work in pip. Maybe it would be great if it gets accepted as standard.

[dimbleby]

@potiuk I've suggested two non-ambiguous and spec-defined ways to express what you're aiming for - what do you think is wrong with them?

(it would be more poetry-friendly to choose the one that doesn't expose a poetry bug, though I think the buggy one is actually what you ought to have intended all along)

[r-richmond]

I think simply currently there is no very well and non-ambigious way to say "depend on any pre-release version of this package but minimum is x.y.z.".

(it would be more poetry-friendly to choose the one that doesn't expose a poetry bug, though I think the buggy one is actually what you ought to have intended all along)

>=1.3.0rc1 (or perhaps to provide more flexibility for earlier development >=1.3.0a1) seems to be the suggestion for airflow to use then if I've followed correctly? I guess the only question left from airflow's side is does pip support that.

[potiuk]

@potiuk I've suggested two non-ambiguous and spec-defined ways to express what you're aiming for - what do you think is wrong with them?

• ==1.3.0.* will not match 1.3.1, 1.4.0 etc. I want to express simple statement "This package require a new feature in depending package that is going to be added in 1.3.0 when released and will be present in any future version as well". I do not want to tie my dependent package to 1.3.0. I want to tell that feature I need is added in 1.3.0 and when I have 1.3.0.a1, 1.3.0.b1, 1.3.0.rc1 but also 1.3.1rc1, 1.3.1, 2.1.0rc1, 2.2.0 and anything like that, it should work

• >=1.3.0rc1 will not work for 1.3.0a1, 1.3.0b2 etc. (neither >=1.3.0a0 for rc and other releases). Each of those (alpha, beta, etc.) are completely separate naming space. Yet my requirement is "the feature is going to be released in 1.3.0, and I want to be able to say my package depends on any alpha, beta, rc and also "official" version of the dependent package. This is what >=1.3.0.* does (and nicely expresses as well).

[dimbleby]

Each of those (alpha, beta, etc.) are completely separate naming space.

this is not true. Their order is well defined by PEP440 and a regular >= starting at the earliest should meet your need.

[potiuk]

this is not true. Their order is well defined by PEP440 and a regular >= starting at the earliest should meet your need.

Citation needed. I have not found such statement in PEP440 (and a) I searched hard b) PyPa members who I talked to confirmed otherwise - cc @uranusjr - unless I misunderstood it of course.

[dimbleby]

https://peps.python.org/pep-0440/#summary-of-permitted-suffixes-and-relative-ordering

Within a numeric release (1.0, 2.7.3), the following suffixes are permitted and MUST be ordered as shown:

.devN, aN, bN, rcN, , .postN

[potiuk]

I belive (and testted it) pip does not work this way @uranusjr - did I misunderstand that - can I use >=1.3.0dev0 and it will cover all alphas/beta/rc releses as well (because if that is correct understanding then the right solution is simply >=1.3.0dev0

[potiuk]

(let me try it). BTW. Will Poetry will work if I put it this way @dimbleby ?

[dimbleby]

both poetry add "apache-airflow-providers-common-sql>=1.3.0dev0 and pip install "apache-airflow-providers-common-sql>=1.3.0dev0" find and install apache-airflow-providers-common-sql-1.3.0rc1

[potiuk]

Yep. confirmed. I must have misunderstood it.

Cool. From my point of view this issue is fixed. Fix is coming to the future RCs if Airlfow momentarily. Thanks for useful discssion.

[dimbleby]

no worries. btw i just noticed that it should be .dev0 (with the dot) and not dev0, but the conclusion is otherwise the same

[potiuk]

Yep. That's what I have in my PR.

[potiuk]

Fix for airflow provider packages in apache/airflow#27727

[r-richmond]

Well TY @potiuk & @dimbleby . My motivation for opening this issue has now been resolved however @dimbleby should we leave this issue open for the issue you found or open a fresh issue with your example (and get a clean conversation history) ?

[dimbleby]

A new one would be lovely, thanks

[potiuk]

And a bit off-topic - any chance on poetry supporting constraints :) @dimbleby ? I would LOVE to be able to tell our users they can use poetry, but lack of constraint support makes it impossible - there are many past cases where airflow stopped to install cleanly because of (even transitive) 3rd-party dependencies and our constraints support saved us from huge problems that we used to have in the past.

[dimbleby]

When you talk about poetry "supporting constraints" - I think you mean: things that are like requirements eg "foo>=1.2", but which don't actually require the named package to be installed?

presumably the use case for such things would be for transitive dependencies where you don't want to specify an explicit dependency, but if the solver does encounter that package then you want to guide it towards (or away from) some particular version?

I've seen requests for similar, but don't recall anything spelling out this particular approach.

Feel free to raise a feature request where folk can figure out whether this is a good idea, or a duplicate, or neither, or both.

[potiuk]

There've been a feature request already with linked PR which I commented on and explained why we need it :) #3225 and it has been closed.

But sure, I can open one again.

[dimbleby]

looks like that one was closed on the grounds that "We already have a lock file, which is effectively what the cosntratint file is". So if you're willing to try again, I guess my advice would be to make clear how this is not just a lock file!

[potiuk]

Created #7051

[r-richmond]

excluding and including pre-releases is a separate question than whether or not the version satisfies the constraint

eg >=1.3.0 is satisfied by 1.3.1rc1 but not by 1.3.0rc1. Only once that standard is met can you ask whether or not the 1.3.1rc pre-release is allowed as a pre-release.

Ah I see, that is not how I read the spec but I can follow that logic.

Unfortunately I don't see any examples listed in pep440 that explicitly disambiguate this situation :S Is there another PEP worth referencing here?

@dimbleby for what it is worth I asked on discuss.python and a couple core maintainers confirmed your understanding of the spec was the correct one and mine was mistaken. cheers!

p.s. Though I did find the following quotes amusing

it seems like pip’s implementation of PEP 440 is already known to be somewhat faulty regarding prereleases.

This matches my intuition when reasoning about versions, regardless of what the tools currently do.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.