-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Adjust limits when constructing cross-provider dependencies #25364
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5a9f742 to
6106414
Compare
Member
Author
Member
Author
|
It just mentions |
When we are constructing cross-provider dependencies, we should adjust the limits to account for some apache-airflow packages that are not yet released. The adjustment is to add ".*" after the version number when dependency is lower-bound with `>=`. It's not explicitly mentioned in PEP 440 - it is only mentioned there that it works for equality, but since `>=` is also part of `equal` it also works there. Example is a common-sql package that might be limited to `>=1.1.0` in google provider as part of the change, but it might not yet be released as it is being released together with the package it is needed by. We need to adjust the version whenever in our providers we refer to other providers with >= install clause because `--pre` flag in `pip` only allows to install direct pre-release (and development) dependencies but it does not modify requirements of those package to also include pre-releases as transitive dependency. Also, we need to remove the limits in `devel` dependencies because the packages are not yet released at the moment we use those dependencies, so the limits in `devel` should be removed, allowing the developers to install Airflow without those devel packages. Also a bug was found that would prevent to generate the dependencies in case provider.yaml file only changed (bad specification of .pre-commit include)
6106414 to
5605182
Compare
Member
Author
eladkal
approved these changes
Jul 29, 2022
potiuk
added a commit
that referenced
this pull request
Aug 4, 2022
When we are constructing cross-provider dependencies, we should adjust the limits to account for some apache-airflow packages that are not yet released. The adjustment is to add ".*" after the version number when dependency is lower-bound with `>=`. It's not explicitly mentioned in PEP 440 - it is only mentioned there that it works for equality, but since `>=` is also part of `equal` it also works there. Example is a common-sql package that might be limited to `>=1.1.0` in google provider as part of the change, but it might not yet be released as it is being released together with the package it is needed by. We need to adjust the version whenever in our providers we refer to other providers with >= install clause because `--pre` flag in `pip` only allows to install direct pre-release (and development) dependencies but it does not modify requirements of those package to also include pre-releases as transitive dependency. Also, we need to remove the limits in `devel` dependencies because the packages are not yet released at the moment we use those dependencies, so the limits in `devel` should be removed, allowing the developers to install Airflow without those devel packages. Also a bug was found that would prevent to generate the dependencies in case provider.yaml file only changed (bad specification of .pre-commit include) (cherry picked from commit 5e423f5)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When we are constructing cross-provider dependencies, we should adjust
the limits to account for some apache-airflow packages that are not yet
released.
The adjustment is to add ".*" after the version number when
dependency is lower-bound with
>=. It's not explicitly mentioned inPEP 440 - it is only mentioned there that it works for
equality, but since
>=is also part ofequalit alsoworks there.
Example is a common-sql package that might be limited to
>=1.1.0in google provider as part of the change, but it might not yet be released
as it is being released together with the package it is needed by.
We need to adjust the version whenever in our providers we
refer to other providers with >= install clause because
--preflag in
piponly allows to install direct pre-release (anddevelopment) dependencies but it does not modify requirements of
those package to also include pre-releases as transitive dependency.
Also, we need to remove the limits in
develdependencies becausethe packages are not yet released at the moment we use those
dependencies, so the limits in
develshould be removed, allowingthe developers to install Airflow without those devel packages.
Also a bug was found that would prevent to generate
the dependencies in case provider.yaml file only changed (bad
specification of .pre-commit include)
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.