Add Airflow 3.2 in test matrix by pankajastro · Pull Request #2472 · astronomer/astronomer-cosmos

pankajastro · 2026-03-17T16:10:15Z

Add Airflow 3.2 to the Cosmos test matrix and resolve all compatibility issues.

Changes

Airflow 3.2 compatibility

Add Airflow 3.2 to the CI test matrix (.github/workflows/test.yml, pyproject.toml)
Add pinned requirements-airflow-3.2-dbt-1.11.txt with versions resolved for Python
3.10 compatibility across all supported Python versions

Import path changes for Airflow 3.2

airflow.utils.context.Context → airflow.sdk.definitions.context.Context
airflow.utils.context.context_merge → airflow.sdk.definitions.context.context_merge
airflow.datasets.DatasetAlias → airflow.sdk.definitions.asset.AssetAlias
airflow.models.dagbag.DagBag, sync_bag_to_db → airflow.dag_processing.dagbag
airflow.configuration.conf → airflow.providers.common.compat.sdk.conf (per
Update conf imports in providers to use airflow.sdk.configuration apache/airflow#60000)

All import changes use try/except to maintain backward compatibility with older Airflow
versions.

Other fixes

Handle Airflow 3.2 WeightRule API change in TaskInstance creation (tests/conftest.py)
Handle SDK AirflowConfigException in cosmos/settings.py (different exception class in
Airflow 3.2)

closes: #2403

Co-authored-by: Pankaj Koti pankajkoti699@gmail.com

codecov · 2026-03-17T17:26:25Z

Codecov Report

❌ Patch coverage is 98.03922% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 98.07%. Comparing base (331b90a) to head (402c4e2).

Files with missing lines	Patch %	Lines
cosmos/__init__.py	95.45%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2472      +/-   ##
==========================================
+ Coverage   97.97%   98.07%   +0.09%     
==========================================
  Files         103      103              
  Lines        7455     7465      +10     
==========================================
+ Hits         7304     7321      +17     
+ Misses        151      144       -7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…st DAGs Resolve merge conflicts with main in cosmos/__init__.py (keep lazy settings import to avoid circular import on Airflow 3.2, use version from main) and tests/test_example_dags.py (take main's version). Upgrade Airflow 3.2 from beta2 to rc2 which contains the ObjectStoragePath credential resolution fix. Remove the redundant 3.2.0b2 special-case in integration-setup.sh. Remove the Airflow >3.1 .airflowignore entries for cosmos_manifest_example.py, cosmos_manifest_selectors_example.py, and cross_project_manifest_dag.py from both test_example_dags.py and test_example_dags_no_connections.py so these DAGs are tested in CI again. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Airflow 3.2 changed the initialization order of airflow.configuration, causing circular imports when providers import conf from it during startup. The official migration path (apache/airflow#60000) is to use the compat SDK import instead. Add type annotations for settings variables used by cache.py since the compat conf lacks the same type stubs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Airflow 3.2 compat conf raises airflow.sdk._shared.configuration.exceptions.AirflowConfigException instead of airflow.exceptions.AirflowConfigException. Catch the SDK exception class when available. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace eager imports with lazy loading via module-level __getattr__. Imports are deferred until first access, which: - Avoids circular imports during Airflow >= 3.2 provider discovery where airflow.configuration is not yet fully initialized - Reduces memory footprint by only loading modules that are used - Makes the enable_memory_optimised_imports setting unnecessary Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Needed for the cross_project_manifest_dag example DAG. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace multiple uv pip install calls with a single pinned requirements file for Airflow 3.2, matching the pattern used by other Airflow versions. Generated via uv pip compile to ensure dependency consistency. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Remove numpy, scikit-learn, scipy, pandas, db-dtypes, google-cloud-aiplatform, networkx, and paradime-io pins that have Python-version-specific constraints. These are transitive dependencies resolved differently per Python version. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace rc2 with stable 3.2.0 in requirements and remove the --prerelease=allow flag from the pre-install script. Regenerate the requirements file without pre-release packages. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Pin numpy, scikit-learn, scipy, pandas, networkx, paradime-io, db-dtypes, google-cloud-aiplatform, joblib, and threadpoolctl using versions resolved for Python 3.10, which are compatible across all supported Python versions (3.10-3.13). Also fix RST table alignment in scheduling docs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR adds Apache Airflow 3.2 to the project’s CI/test matrix and updates Cosmos + tests to remain compatible across Airflow 2.x–3.2, mainly by handling Airflow 3.2 import-path and API changes via backward-compatible fallbacks.

Changes:

Add Airflow 3.2 to CI/hatch test matrices and add a pinned requirements set for Airflow 3.2 + dbt 1.11.
Update Airflow imports (Context/context_merge/DagBag/DatasetAlias/conf) with try/except fallbacks for cross-version compatibility.
Adjust tests/utilities to handle Airflow 3.2 TaskInstance/WeightRule differences via a shared helper.

Reviewed changes

Copilot reviewed 21 out of 22 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/utils.py	Use new Airflow 3.2 DagBag import path with fallback to legacy location.
tests/operators/test_local.py	Switch to shared TaskInstance factory and update Context import for Airflow 3.2+.
tests/operators/test_kubernetes.py	Switch to shared TaskInstance factory and update Context import for Airflow 3.2+.
tests/operators/test_gcp_cloud_run_job.py	Update Context import for Airflow 3.2+ compatibility.
tests/operators/test_docker.py	Update Context import for Airflow 3.2+ compatibility.
tests/operators/test_base.py	Update Context import for Airflow 3.2+ compatibility.
tests/operators/test_azure_container_instance.py	Update Context import for Airflow 3.2+ compatibility.
tests/operators/test_aws_ecs.py	Update Context import for Airflow 3.2+ compatibility.
tests/conftest.py	Add `make_task_instance` helper to handle Airflow 3.2 WeightRule/TaskInstance behavior.
scripts/test/pre-install-airflow.sh	Add install branch for Airflow 3.2 pinned requirements.
scripts/test/integration-setup.sh	Harden Airflow version parsing and log detected version.
requirements/requirements-airflow-3.2-dbt-1.11.txt	Add fully pinned dependency set for Airflow 3.2 + dbt 1.11.
pyproject.toml	Add Airflow 3.2 to hatch test matrix.
docs/index.rst	Fix indentation in the main docs toctree.
docs/guides/run_dbt/customization/scheduling.rst	Adjust table formatting in scheduling guide.
cosmos/settings.py	Use compat `conf` import and handle SDK config exception differences in Airflow 3.2.
cosmos/plugin/airflow3.py	Use compat `conf` import for Airflow 3.2+.
cosmos/operators/kubernetes.py	Update `context_merge` import to Airflow 3.2 SDK location with fallback.
cosmos/operators/base.py	Update `context_merge` import to Airflow 3.2 SDK location with fallback.
cosmos/operators/_asynchronous/bigquery.py	Handle DatasetAlias→AssetAlias import move with fallback for older Airflow.
cosmos/init.py	Bump package version to 1.14.0a7.
.github/workflows/test.yml	Add Airflow 3.2 to CI matrices and adjust workflow push branch trigger.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Fix RST table column alignment in scheduling.rst and fix toctree indentation in index.rst. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

tatiana

Thanks @pankajastro for starting the work, and @pankajkoti for taking it over - Airlfow 3.2.0 was just released, and it will be great to have Cosmos official support to it.

I'm pre-approving the PR, and the merge is subject to the tests passing.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

This reverts commit e314270.

Copilot

Pull request overview

Copilot reviewed 21 out of 22 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Add try/except for Context import in bigquery.py and databricks.py to handle Airflow 3.2 where Context moved to airflow.sdk. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

pankajkoti · 2026-04-07T16:14:33Z

All tests have passed https://github.com/astronomer/astronomer-cosmos/actions/runs/24088244912?pr=2472 except for the Kubernetes test that appears to be remaining stuck after the merge of PR #2529.

Hence, I am going ahead with the merge.

Copilot

Pull request overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-07T16:21:39Z

    _read_target_sources_json,
 )
 from cosmos.profiles import PostgresUserPasswordProfileMapping
+from tests.conftest import make_task_instance


Avoid importing from tests.conftest in test modules: pytest loads conftest.py as a special plugin module (typically under the name conftest), so importing it as tests.conftest can execute the file a second time under a different module name. Consider moving make_task_instance into a dedicated helper module (e.g., tests/utils_task_instance.py or tests/utils.py) and importing it from there; conftest.py can also import/re-export it if needed.

Suggested change

from tests.conftest import make_task_instance

from conftest import make_task_instance

Copilot · 2026-04-07T16:21:40Z

    DbtTestWarningHandler,
 )
 from cosmos.profiles import PostgresUserPasswordProfileMapping
+from tests.conftest import make_task_instance


Avoid importing from tests.conftest in test modules: pytest loads conftest.py as a special plugin module (typically under the name conftest), so importing it as tests.conftest can execute the file a second time under a different module name. Consider moving make_task_instance into a dedicated helper module (e.g., tests/utils_task_instance.py or tests/utils.py) and importing it from there; conftest.py can also import/re-export it if needed.

Suggested change

from tests.conftest import make_task_instance

from conftest import make_task_instance

Add Airflow 3.2 in test matrix

3836c35

pankajastro temporarily deployed to internal March 17, 2026 16:10 — with GitHub Actions Inactive

Update test script to handle AF3.2 beta

3636c12

pankajastro temporarily deployed to internal March 17, 2026 16:15 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal March 17, 2026 16:16 — with GitHub Actions Inactive

Update test script to handle AF3.2 beta

51d7444

pankajastro temporarily deployed to internal March 17, 2026 16:19 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal March 17, 2026 16:20 — with GitHub Actions Inactive

Update test script to handle AF3.2 beta

8852da4

pankajastro temporarily deployed to internal March 17, 2026 16:24 — with GitHub Actions Inactive

Update test script to handle AF3.2 beta

45184d5

pankajastro temporarily deployed to internal March 17, 2026 16:26 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal March 17, 2026 16:27 — with GitHub Actions Inactive

Fix import

471c673

pankajastro temporarily deployed to internal March 17, 2026 16:36 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal March 17, 2026 16:37 — with GitHub Actions Inactive

Fix import

d2ea0ed

pankajastro temporarily deployed to internal March 17, 2026 16:38 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal March 17, 2026 16:39 — with GitHub Actions Inactive

Fix import

eeab870

pankajastro temporarily deployed to internal March 17, 2026 18:11 — with GitHub Actions Inactive

Fix import

909a474

pankajastro temporarily deployed to internal March 17, 2026 18:41 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal March 17, 2026 18:42 — with GitHub Actions Inactive

Fix unit tests

6fcefa7

pankajastro temporarily deployed to internal March 18, 2026 11:51 — with GitHub Actions Inactive

pip some versions

c57f2ad

pankajkoti mentioned this pull request Apr 2, 2026

Add support to Airflow 3.2.0 #2403

Closed

pankajkoti and others added 16 commits April 6, 2026 14:58

Merge branch 'main' into airflow_3.2

2040450

Merge branch 'main' into airflow_3.2

a569aab

Add dbt-loom to Airflow 3.2 requirements

d2b451d

Needed for the cross_project_manifest_dag example DAG. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Release 1.14 alpha7

8f34ee7

Install dbt-loom via pre-install script

402c4e2

Merge branch 'main' into airflow_3.2

883c475

Merge branch 'main' into airflow_3.2

20732ff

Merge branch 'main' into airflow_3.2

e1d409c

Update to stable Airflow 3.2.0 release

980b69a

Replace rc2 with stable 3.2.0 in requirements and remove the --prerelease=allow flag from the pre-install script. Regenerate the requirements file without pre-release packages. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

pankajkoti reviewed Apr 7, 2026

View reviewed changes

Comment thread .github/workflows/test.yml Outdated

Copilot AI reviewed Apr 7, 2026

View reviewed changes

Comment thread tests/operators/test_local.py

Comment thread .github/workflows/test.yml

Fix docs build warnings treated as errors (#2532)

74b62db

Fix RST table column alignment in scheduling.rst and fix toctree indentation in index.rst. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

tatiana approved these changes Apr 7, 2026

View reviewed changes

pankajkoti and others added 2 commits April 7, 2026 20:16

Update tests/operators/test_local.py

e314270

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Revert "Update tests/operators/test_local.py"

4fe08cd

This reverts commit e314270.

Copilot AI reviewed Apr 7, 2026

View reviewed changes

Comment thread cosmos/operators/_asynchronous/bigquery.py

Comment thread tests/operators/test_local.py

Fix Context import for Airflow 3.2 in async operators

f42835d

Add try/except for Context import in bigquery.py and databricks.py to handle Airflow 3.2 where Context moved to airflow.sdk. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Update .github/workflows/test.yml

0530892

Copilot AI reviewed Apr 7, 2026

View reviewed changes

pankajastro mentioned this pull request Apr 7, 2026

Release 1.14.0 #2314

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Airflow 3.2 in test matrix#2472

Add Airflow 3.2 in test matrix#2472
pankajkoti merged 61 commits into
mainfrom
airflow_3.2

pankajastro commented Mar 17, 2026 •

edited by pankajkoti

Loading

Uh oh!

codecov Bot commented Mar 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

pankajkoti commented Apr 7, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 7, 2026

Uh oh!

Copilot AI Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	from tests.conftest import make_task_instance
	from conftest import make_task_instance

Conversation

pankajastro commented Mar 17, 2026 • edited by pankajkoti Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Airflow 3.2 compatibility

Import path changes for Airflow 3.2

Other fixes

Uh oh!

codecov Bot commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

pankajkoti commented Apr 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pankajastro commented Mar 17, 2026 •

edited by pankajkoti

Loading

codecov Bot commented Mar 17, 2026 •

edited

Loading