Fix broken CI tests.py3.11-2.8-1.9:test-integration-setup#1902
Merged
Conversation
Installing openlineage-airflow was leading to AF3 to be installed instead of AF2.8 when running ```bash hatch run tests.py3.11-2.8-1.9:test-integration-setup ```
✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.
|
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR fixes a CI test issue where installing openlineage-airflow was inadvertently upgrading Apache Airflow from version 2.8 to version 3.0, breaking the test environment. The fix ensures that Airflow version constraints are properly enforced during package installation.
- Moved openlineage-airflow installation to the pre-install script with proper version constraints
- Added Airflow version pinning when installing openlineage-airflow to prevent upgrades
- Added debugging output and cleanup steps to improve test reliability
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| scripts/test/pre-install-airflow.sh | Added openlineage-airflow installation with Airflow version pinning for versions 2.4-2.6, plus debugging output |
| scripts/test/integration-setup.sh | Removed openlineage-airflow installation and added cleanup of Airflow config/database files |
| scripts/test/integration.sh | Added specific test file to the pytest command |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
tatiana
changed the title
tests.py3.11-2.8-1.9:test-integration-setup
tatiana
commented
Aug 5, 2025
tatiana
force-pushed
the
fix-broken-ci-test
branch
from
92e3619 to
b125896
Compare
tatiana
commented
Aug 5, 2025
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1902 +/- ##
=======================================
Coverage 98.06% 98.06%
=======================================
Files 85 85
Lines 5312 5312
=======================================
Hits 5209 5209
Misses 103 103 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
pankajkoti
approved these changes
Aug 6, 2025
Contributor
pankajkoti
left a comment
pankajkoti
left a comment
There was a problem hiding this comment.
Nice catch and fix! Thanks!
Merged
tatiana
added a commit
that referenced
this pull request
Aug 8, 2025
Installing `openlineage-airflow` and `airflow-provider-duckdb` without restricting the version of Airflow was leading to AF3 to be installed instead of AF2.8 when running ```bash hatch run tests.py3.11-2.8-1.9:test-integration-setup ``` As can be seen in this GH action CI job run: https://github.com/astronomer/astronomer-cosmos/actions/runs/16731639151/job/47426561634?pr=1901 Which led other PRs to fail: #1901 Closes: #1905 (cherry picked from commit ae453d5)
tatiana
added a commit
that referenced
this pull request
Aug 8, 2025
**Bug Fixes** * Fix task instance ``try_number`` attribute for Airflow 3 compatibility by @pankajkoti in #1781 * Fix rendered template override logic when ``should_store_compiled_sql=False`` to restore pre-refactor behaviour by @pankajkoti in #1777 * Fix ``ProfileConfig`` in GCP Cloud Run job execution mode by @ramonvermeulen in #1783 * Fix dbt Docs page height by @1cadumagalhaes in #1793 * Add support to base64 encoded pem in Snowflake profiles by @brunocmartins in #1801 * Allow to disable owner inheritance from dbt into airflow DAG owners by @CorsettiS in #1787 * Fix Kubernetes Pod Operator conversion of ``container_resources`` to ``resources`` by @johnhoran in #1821 * Fix ``dbt deps`` with project level variables by @AlexandrKhabarov in #1822 * Fix source freshness warnings in kubernetes execution mode by @Pawel-Drabczyk in #1859 * Fix: Harden DbtNode against null config/meta by @pankajkoti in #1877 * Fix cache behaviour when DAG name contains "." by @tatiana in #1908 **Documentation** * Fix ``contributing.rst`` docs by @tatiana in #1785 * Fix docs rendering in Airflow 3 Compatibility by @pankajastro in #1790 * Fix typo in ``selecting-excluding.rst`` by @msshroff in #1814 * Update testing behavior file with ``ExecutionMode.KUBERNETES`` by @LuigiCerone in #1813 * Add step to fork repo in contributing guide by @pankajastro in #1808 * Fix ``depends_on`` attribute by @benedikt-buchert in #1837 * Fix character name by @ThePsyjo in #1860 * Update suggested MWAA startup script by @jaklan in #1884 * Make implementation & docs consistent regarding ``use_dataset_airflow3_uri_standard`` by @Anti0ff in #1878 **Others** * Set retries to 0 in example DAGs by @pankajkoti in #1782 * Fix ``test_async_example_dag_without_setup_task`` tests by @pankajastro in #1788 * Fix test hash value for Darwin when using Py 3.12.10 by @tatiana in #1786 * Upgrade Python and Airflow used to run MyPy checks by @tatiana in #1796 * Assert example DAGs' ``DagRunState`` and fix issues by @pankajkoti and @tatiana in #1778 * Update the conflict matrix to include AF 2.10, 2.11 & 3.0 and dbt 1.9 & 1.10 by @tatiana in #1820 * Fix broken CI due to Pydantic conflicts by @tatiana in #1809 * Drop Python 3.8 Support by @pankajastro in #1852 * Add Airflow 2.11 to the test matrix by @tatiana in #1807 * Require Authorize for all jobs on pull requests from external contributors in CI by @pankajkoti in #1861 * Leverage Trusted Publisher Management when publishing PyPI package by @tatiana in #1862 * CI: Add back accidentally deleted python-version matrix for running unit tests by @pankajkoti in #1872 * Remove commented code and fix mypy failures by @pankajkoti in #1876 * Add Zizmor analysis GitHub action by @pankajkoti in #1870 * Catch FlushError on Datasets for Airflow 2.11 dags test by @pankajkoti in #1880 * Add deprecation warning for ``LoadMode.CUSTOM`` parser by @duongphannamhung in #1885 * CI: Add GitHub CodeQL analysis workflow (codeql.yml) by @pankajkoti in #1871 * Resolve 'credential persistence through GitHub Actions artifacts' warnings from Zizmor by @pankajkoti in #1890 * Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti in #1889 * Resolve Zizmor error alerts for unpinned image references; mark alert for pull_request_target ignored by @pankajkoti in #1888 * Fix broken CI ``tests.py3.11-2.8-1.9:test-integration-setup`` by @tatiana in #1902 * Add dbt-core 1.10 to test matrix by @tatiana in #1767 * Pin package dbt-databricks by @pankajastro in #1909 * Enable matrix test entry for dbt-1.9, python-3.9 and airflow-3.0 tests in CI by @pankajastro in #1900 * Pre-commit updates: #1779, #1795, #1800, #1857, #1863, #1869, #1892, #1901 * Dependabot updates: #1904 Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Merged
tatiana
added a commit
that referenced
this pull request
Aug 8, 2025
**Bug Fixes** * Fix task instance ``try_number`` attribute for Airflow 3 compatibility by @pankajkoti in #1781 * Fix rendered template override logic when ``should_store_compiled_sql=False`` to restore pre-refactor behaviour by @pankajkoti in #1777 * Fix ``ProfileConfig`` in GCP Cloud Run job execution mode by @ramonvermeulen in #1783 * Fix dbt Docs page height by @1cadumagalhaes in #1793 * Add support to base64 encoded pem in Snowflake profiles by @brunocmartins in #1801 * Allow to disable owner inheritance from dbt into airflow DAG owners by @CorsettiS in #1787 * Fix Kubernetes Pod Operator conversion of ``container_resources`` to ``resources`` by @johnhoran in #1821 * Fix ``dbt deps`` with project level variables by @AlexandrKhabarov in #1822 * Fix source freshness warnings in kubernetes execution mode by @Pawel-Drabczyk in #1859 * Fix: Harden DbtNode against null config/meta by @pankajkoti in #1877 * Fix cache behaviour when DAG name contains "." by @tatiana in #1908 **Documentation** * Fix ``contributing.rst`` docs by @tatiana in #1785 * Fix docs rendering in Airflow 3 Compatibility by @pankajastro in #1790 * Fix typo in ``selecting-excluding.rst`` by @msshroff in #1814 * Update testing behavior file with ``ExecutionMode.KUBERNETES`` by @LuigiCerone in #1813 * Add step to fork repo in contributing guide by @pankajastro in #1808 * Fix ``depends_on`` attribute by @benedikt-buchert in #1837 * Fix character name by @ThePsyjo in #1860 * Update suggested MWAA startup script by @jaklan in #1884 * Make implementation & docs consistent regarding ``use_dataset_airflow3_uri_standard`` by @Anti0ff in #1878 **Others** * Set retries to 0 in example DAGs by @pankajkoti in #1782 * Fix ``test_async_example_dag_without_setup_task`` tests by @pankajastro in #1788 * Fix test hash value for Darwin when using Py 3.12.10 by @tatiana in #1786 * Upgrade Python and Airflow used to run MyPy checks by @tatiana in #1796 * Assert example DAGs' ``DagRunState`` and fix issues by @pankajkoti and @tatiana in #1778 * Update the conflict matrix to include AF 2.10, 2.11 & 3.0 and dbt 1.9 & 1.10 by @tatiana in #1820 * Fix broken CI due to Pydantic conflicts by @tatiana in #1809 * Drop Python 3.8 Support by @pankajastro in #1852 * Add Airflow 2.11 to the test matrix by @tatiana in #1807 * Require Authorize for all jobs on pull requests from external contributors in CI by @pankajkoti in #1861 * Leverage Trusted Publisher Management when publishing PyPI package by @tatiana in #1862 * CI: Add back accidentally deleted python-version matrix for running unit tests by @pankajkoti in #1872 * Remove commented code and fix mypy failures by @pankajkoti in #1876 * Add Zizmor analysis GitHub action by @pankajkoti in #1870 * Catch FlushError on Datasets for Airflow 2.11 dags test by @pankajkoti in #1880 * Add deprecation warning for ``LoadMode.CUSTOM`` parser by @duongphannamhung in #1885 * CI: Add GitHub CodeQL analysis workflow (codeql.yml) by @pankajkoti in #1871 * Resolve 'credential persistence through GitHub Actions artifacts' warnings from Zizmor by @pankajkoti in #1890 * Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti in #1889 * Resolve Zizmor error alerts for unpinned image references; mark alert for pull_request_target ignored by @pankajkoti in #1888 * Fix broken CI ``tests.py3.11-2.8-1.9:test-integration-setup`` by @tatiana in #1902 * Add dbt-core 1.10 to test matrix by @tatiana in #1767 * Pin package dbt-databricks by @pankajastro in #1909 * Enable matrix test entry for dbt-1.9, python-3.9 and airflow-3.0 tests in CI by @pankajastro in #1900 * Pre-commit updates: #1779, #1795, #1800, #1857, #1863, #1869, #1892, #1901 * Dependabot updates: #1904 Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
tatiana
added a commit
that referenced
this pull request
Feb 11, 2026
**Context** Over the past few months, we have experienced recurring CI failures caused by dependency conflicts. These issues are not triggered by changes in Cosmos itself, but rather by unpinned and dynamically resolved dependencies. These conflicts have been disruptive, time-consuming to debug, and difficult to reproduce locally. Some recent examples: - #2268 - #2172 - #2076 - #1987 - #1909 - #1902 **Current Situation** We have tried several approaches to mitigate dependency issues: - Dynamically resolving dependencies while restricting requirements as little as possible - Handling Airflow version–specific dependencies individually - Sometimes relying on Airflow constraints (which do not include provider versions), sometimes not - Attempting to isolate dependencies in `pre-install-airflow.sh` More recently, however, Airflow began being reinstalled via `integration-setup.sh`, leading to overlapping installations and Python virtual environments with conflicting packages. Additionally, if we try to recreate the environment locally using the exact dependency list printed by the CI at the end of the “Install packages and dependencies” step, the environment cannot be created due to dependencies conflicts. This results in: - Non-deterministic Python package installation - CI failures that are difficult (or impossible) to reproduce locally - Significant frustration and lost engineering time **Proposed Approach** This PR proposes pinning dependencies more strictly to make our CI environments deterministic and reproducible. I do not believe this is a perfect solution. Some trade-offs include: - We will no longer automatically test against newly released dependency versions - Pinned versions may require manual updates, including potential security updates This PR also does not introduce pip constraints files (since we currently rely on pip install -r). However, it can be viewed as a step toward the approach discussed in: #967 I discussed this proposal with @pankajastro and @pankajkoti, and we agreed it is worth trying. If this strategy does not work well in practice, we can revert to the current approach. **Known Side Effects** The changes in this PR introduce a few limitations: - Tests no longer run the cross-project example DAG (dbt Loom) using the LoadMode.DBT_LS approach, as dbt Loom is now installed in a separate virtual environment due to dependency conflicts. - The S3 load manifest task is no longer executed in the corresponding DAG for Airflow 2.9, due to conflicts between aiobotocore and other dependencies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Installing
openlineage-airflowandairflow-provider-duckdbwithout restricting the version of Airflow was leading to AF3 to be installed instead of AF2.8 when runningAs can be seen in this GH action CI job run:
https://github.com/astronomer/astronomer-cosmos/actions/runs/16731639151/job/47426561634?pr=1901
Which led other PRs to fail:
#1901
Closes: #1905