Skip to content

Resolve CI failures for Apache Airflow 2.7 jobs #1182

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pankajkoti merged 12 commits into from
Aug 29, 2024
Merged

Resolve CI failures for Apache Airflow 2.7 jobs #1182

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
86fcb1f
Comment the dependencies temporarily causing deep resolution errors
pankajkoti Aug 27, 2024
1ffcdc1
Try next combination
pankajkoti Aug 27, 2024
1d9edfb
Comment amazon provider dependency
pankajkoti Aug 28, 2024
e8dd868
Comment AWS EKS tests
pankajkoti Aug 28, 2024
1fab2df
Restrict amazon provider package to <8.28
pankajkoti Aug 28, 2024
82f9b69
Use Airflow 2.8 instead of 2.7 for ancillary jobs
pankajkoti Aug 28, 2024
5569497
Run workflow on branch
pankajkoti Aug 28, 2024
26d3833
Use minified constraints style install of airflow providers
pankajkoti Aug 28, 2024
e4f0898
Undo commented out code in DAGs & tests
pankajkoti Aug 28, 2024
5401e6c
Install with constraints
pankajkoti Aug 28, 2024
f059ab2
Exclude Python 3.8 & 3.9 with Apache Airflow 2.7 in unit & integratio…
pankajkoti Aug 29, 2024
5af98d7
Update .github/workflows/test.yml
pankajkoti Aug 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ jobs:
architecture: "x64"

- run: pip3 install hatch
- run: hatch run tests.py3.9-2.7:type-check
- run: hatch run tests.py3.9-2.8:type-check

Run-Unit-Tests:
runs-on: ubuntu-latest
Expand All @@ -59,6 +59,12 @@ jobs:
airflow-version: "2.7"
- python-version: "3.12"
airflow-version: "2.8"
# It's observed that the dependencies resolution for Apache Airflow versions 2.7 are error-ring out with deep
# resolutions. This is a temporary exclusion until the issue is resolved.
- python-version: "3.8"
airflow-version: "2.7"
- python-version: "3.9"
airflow-version: "2.7"
steps:
- uses: actions/checkout@v3
with:
Expand Down Expand Up @@ -104,6 +110,12 @@ jobs:
airflow-version: "2.4"
- python-version: "3.11"
airflow-version: "2.5"
# It's observed that the dependencies resolution for Apache Airflow versions 2.7 are error-ring out with deep
# resolutions. This is a temporary exclusion until the issue is resolved.
- python-version: "3.8"
airflow-version: "2.7"
- python-version: "3.9"
airflow-version: "2.7"
services:
postgres:
image: postgres
Expand Down Expand Up @@ -254,7 +266,7 @@ jobs:
strategy:
matrix:
python-version: ["3.11"]
airflow-version: ["2.7"]
airflow-version: ["2.8"]

steps:
- uses: actions/checkout@v3
Expand Down Expand Up @@ -319,7 +331,7 @@ jobs:
strategy:
matrix:
python-version: [ "3.11" ]
airflow-version: [ "2.7" ]
airflow-version: [ "2.8" ]
services:
postgres:
image: postgres
Expand Down Expand Up @@ -395,7 +407,7 @@ jobs:
strategy:
matrix:
python-version: ["3.11"]
airflow-version: ["2.7"]
airflow-version: ["2.8"]
num-models: [1, 10, 50, 100]
services:
postgres:
Expand Down
11 changes: 11 additions & 0 deletions scripts/test/pre-install-airflow.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,15 @@ mv /tmp/constraint.txt.tmp /tmp/constraint.txt
pip install uv
uv pip install "apache-airflow==$AIRFLOW_VERSION" --constraint /tmp/constraint.txt
uv pip install pydantic --constraint /tmp/constraint.txt


if [ "$AIRFLOW_VERSION" = "2.7" ]; then
uv pip install "apache-airflow-providers-amazon[s3fs]" --constraint /tmp/constraint.txt
uv pip install "apache-airflow-providers-cncf-kubernetes" --constraint /tmp/constraint.txt
uv pip install "apache-airflow-providers-docker" --constraint /tmp/constraint.txt
uv pip install "apache-airflow-providers-google" --constraint /tmp/constraint.txt
uv pip install "apache-airflow-providers-microsoft-azure" --constraint /tmp/constraint.txt
uv pip install "apache-airflow-providers-postgres" --constraint /tmp/constraint.txt
fi

rm /tmp/constraint.txt