Skip to content

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented Jul 28, 2022

When we are constructing cross-provider dependencies, we should adjust
the limits to account for some apache-airflow packages that are not yet
released.

The adjustment is to add ".*" after the version number when
dependency is lower-bound with >=. It's not explicitly mentioned in
PEP 440 - it is only mentioned there that it works for
equality, but since >= is also part of equal it also
works there.

Example is a common-sql package that might be limited to >=1.1.0
in google provider as part of the change, but it might not yet be released
as it is being released together with the package it is needed by.

We need to adjust the version whenever in our providers we
refer to other providers with >= install clause because --pre
flag in pip only allows to install direct pre-release (and
development) dependencies but it does not modify requirements of
those package to also include pre-releases as transitive dependency.

Also, we need to remove the limits in devel dependencies because
the packages are not yet released at the moment we use those
dependencies, so the limits in devel should be removed, allowing
the developers to install Airflow without those devel packages.

Also a bug was found that would prevent to generate
the dependencies in case provider.yaml file only changed (bad
specification of .pre-commit include)


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk potiuk requested review from josh-fell, kaxil and uranusjr July 28, 2022 11:51
@potiuk potiuk force-pushed the strip-limits-from-devel-all branch 2 times, most recently from 5a9f742 to 6106414 Compare July 28, 2022 11:53
@potiuk
Copy link
Member Author

potiuk commented Jul 28, 2022

I deleted accidentally the branch in #25306 - but this one is refreshed @uranusjr - to reflect the finding from there. I added better description and explanation why we are adding .* and also test it in #24836

@potiuk
Copy link
Member Author

potiuk commented Jul 28, 2022

It just mentions -pre behaviour as it is, without judging if it is a bug or not (because I do not know - it could be intentional or accidental behaviour which simply does not include our case when we need to install pre-release package which depends on specific version other pre-release package that has no released version yet).

When we are constructing cross-provider dependencies, we should adjust
the limits to account for some apache-airflow packages that are not yet
released.

The adjustment is to add ".*" after the version number when
dependency is lower-bound with `>=`. It's not explicitly mentioned in
PEP 440 - it is only mentioned there that it works for
equality, but since `>=` is also part of `equal` it also
works there.

Example is a common-sql package that might be limited to `>=1.1.0`
in google provider as part of the change, but it might not yet be released
as it is being released together with the package it is needed by.

We  need to adjust the version whenever in our providers we
refer to other providers with >= install clause because  `--pre`
flag in `pip` only allows to install direct pre-release (and
development) dependencies but it does not modify requirements of
those package to also include pre-releases as transitive dependency.

Also, we need to remove the limits in `devel` dependencies because
the packages are not yet released at the moment we use those
dependencies, so the limits in `devel` should be removed, allowing
the developers to install Airflow without those devel packages.

Also a bug was found that would prevent to generate
the dependencies in case provider.yaml file only changed (bad
specification of .pre-commit include)
@potiuk potiuk force-pushed the strip-limits-from-devel-all branch from 6106414 to 5605182 Compare July 28, 2022 13:09
@potiuk potiuk changed the title Adjust limits when constructing devel dependencies Adjust limits when constructing cross-provider dependencies Jul 28, 2022
@potiuk
Copy link
Member Author

potiuk commented Jul 29, 2022

@uranusjr @eladkal -> now that the #25350 is green - I would like to merge this one first, so that it is separated from the "move' (I incorporated that one in the SQL move to test that it works).

@potiuk potiuk requested a review from eladkal July 29, 2022 10:19
@potiuk potiuk merged commit 5e423f5 into apache:main Jul 29, 2022
@potiuk potiuk deleted the strip-limits-from-devel-all branch July 29, 2022 11:24
potiuk added a commit that referenced this pull request Aug 4, 2022
When we are constructing cross-provider dependencies, we should adjust
the limits to account for some apache-airflow packages that are not yet
released.

The adjustment is to add ".*" after the version number when
dependency is lower-bound with `>=`. It's not explicitly mentioned in
PEP 440 - it is only mentioned there that it works for
equality, but since `>=` is also part of `equal` it also
works there.

Example is a common-sql package that might be limited to `>=1.1.0`
in google provider as part of the change, but it might not yet be released
as it is being released together with the package it is needed by.

We  need to adjust the version whenever in our providers we
refer to other providers with >= install clause because  `--pre`
flag in `pip` only allows to install direct pre-release (and
development) dependencies but it does not modify requirements of
those package to also include pre-releases as transitive dependency.

Also, we need to remove the limits in `devel` dependencies because
the packages are not yet released at the moment we use those
dependencies, so the limits in `devel` should be removed, allowing
the developers to install Airflow without those devel packages.

Also a bug was found that would prevent to generate
the dependencies in case provider.yaml file only changed (bad
specification of .pre-commit include)

(cherry picked from commit 5e423f5)
@potiuk potiuk added this to the Airflow 2.3.4 milestone Aug 5, 2022
@ephraimbuddy ephraimbuddy added type:misc/internal Changelog: Misc changes that should appear in change log changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) and removed type:misc/internal Changelog: Misc changes that should appear in change log labels Aug 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants