Skip to content

Remove usage of contextmanager in plugins for accessing connections in Airflow >= 3.1.2#2073

Merged
pankajkoti merged 2 commits into
mainfrom
af3-2-plugins-connections
Nov 4, 2025
Merged

Remove usage of contextmanager in plugins for accessing connections in Airflow >= 3.1.2#2073
pankajkoti merged 2 commits into
mainfrom
af3-2-plugins-connections

Conversation

@pankajkoti
Copy link
Copy Markdown
Contributor

This PR adds explicit Airflow version gating and resilient connection handling for the Airflow 3 FastAPI plugin, along with tests and documentation updates. We now support the plugin only on Airflow >= 3.1 (enforced via a runtime check), move FastAPI app initialisation into the plugin’s constructor to avoid import-time failures, and conditionally use the connection_env context manager only on Airflow 3.1.0–3.1.1 (direct connection access on 3.1.2+). Tests cover the version gate, plugin initialisation, conditional connection handling, and functional routes (skipped on Airflow < 3.1). The documentation is updated to clearly state the Airflow >= 3.1 requirement and configuration keys for dbt_docs_projects.

closes: #2061

Comment thread cosmos/plugin/airflow3.py
@codecov
Copy link
Copy Markdown

codecov Bot commented Oct 31, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.78%. Comparing base (67a53af) to head (0e5b407).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2073   +/-   ##
=======================================
  Coverage   97.77%   97.78%           
=======================================
  Files          91       91           
  Lines        5848     5862   +14     
=======================================
+ Hits         5718     5732   +14     
  Misses        130      130           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@pankajkoti pankajkoti marked this pull request as ready for review October 31, 2025 13:50
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds explicit Airflow version gating (>= 3.1) and resilient connection handling for the Cosmos Airflow 3 FastAPI plugin. The implementation includes runtime version checks, conditional use of the connection_env context manager (only for Airflow 3.1.0-3.1.1), and comprehensive test coverage.

Key changes:

  • Enforces Airflow >= 3.1 requirement through runtime checks in plugin initialization and app creation
  • Conditionally uses connection_env context manager only for Airflow 3.1.0-3.1.1 (direct connection access for 3.1.2+)
  • Moves FastAPI app initialization from class-level to constructor to enable version validation before initialization

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
cosmos/plugin/airflow3.py Adds version gating logic, conditional connection handling based on Airflow version, and moves FastAPI app initialization to constructor
tests/plugin/test_plugin_af3.py Adds comprehensive tests for version gating, plugin initialization, conditional connection handling, and marks existing functional tests to skip on Airflow < 3.1
docs/configuration/hosting-docs.rst Documents the Airflow >= 3.1 requirement for the plugin
docs/configuration/cosmos-conf.rst Updates dbt_docs_projects description to clarify Airflow >= 3.1 requirement
docs/airflow3_compatibility/index.rst Adds prominent note about Airflow >= 3.1 requirement for the FastAPI plugin

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/plugin/test_plugin_af3.py Outdated
Comment thread cosmos/plugin/airflow3.py Outdated
Copy link
Copy Markdown
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks excellent, @pankajkoti , thank you very much!
Minor comments inline.

Copy link
Copy Markdown
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, @pankajkoti , thank you!

@pankajkoti pankajkoti merged commit 6a9b0bb into main Nov 4, 2025
81 checks passed
@pankajkoti pankajkoti deleted the af3-2-plugins-connections branch November 4, 2025 10:12
tatiana pushed a commit that referenced this pull request Nov 7, 2025
…n Airflow >= 3.1.2 (#2073)

This PR adds explicit Airflow version gating and resilient connection
handling for the Airflow 3 FastAPI plugin, along with tests and
documentation updates. We now support the plugin only on Airflow >= 3.1
(enforced via a runtime check), move FastAPI app initialisation into the
plugin’s constructor to avoid import-time failures, and conditionally
use the connection_env context manager only on Airflow 3.1.0–3.1.1
(direct connection access on 3.1.2+). Tests cover the version gate,
plugin initialisation, conditional connection handling, and functional
routes (skipped on Airflow < 3.1). The documentation is updated to
clearly state the Airflow >= 3.1 requirement and configuration keys for
dbt_docs_projects.

closes: #2061
@tatiana tatiana mentioned this pull request Nov 7, 2025
@tatiana tatiana added this to the Cosmos 1.11.1 milestone Nov 10, 2025
tatiana added a commit that referenced this pull request Nov 12, 2025
Bug fixes

* Fix ``ExecutionMode.WATCHER`` deadlock in Airflow 3.0 & 3.1 by
@tatiana in #2087
* Fix ``ExecutionMode.AIRFLOW_ASYNC`` ``TaskGroup`` XCom issue by
@tatiana in #2088
* Guard watcher callback exceptions to avoid hanging producer tasks by
@pankajkoti in #2101
* Fix SQL templated field rendering for dynamically mapped tasks in
Airflow 2 by @tatiana in #2119

Enhancements

* Remove usage of contextmanager in plugins for accessing connections in
Airflow >= 3.1.2 by @pankajkoti in #2073

Docs

* Improve ``ExecutionMode.AIRFLOW_ASYNC`` docs by @tatiana in #2103
* Add note about experimenting threads count for the Watcher Execution
mode by @pankajkoti in #2083
* Fix minor documentation formatting issue by @dnskrv in #2098
* Correct example YAML key from ``operator_args`` to ``operator_kwargs``
by @jx2lee in #2091

Others

* Fix broken CI due to fastapi incompatibility with cadwyn for Airflow 3
by @pankajkoti in #2076
* pre-commit autoupdate in #2078, #2104

related:
astronomer/oss-integrations-private#272
@tatiana tatiana mentioned this pull request Nov 12, 2025
tatiana added a commit that referenced this pull request Nov 13, 2025
Bug fixes

* Fix ``ExecutionMode.WATCHER`` deadlock in Airflow 3.0 & 3.1 by
@tatiana in #2087
* Fix ``ExecutionMode.AIRFLOW_ASYNC`` ``TaskGroup`` XCom issue by
@tatiana in #2088
* Guard watcher callback exceptions to avoid hanging producer tasks by
@pankajkoti in #2101
* Fix SQL templated field rendering for dynamically mapped tasks in
Airflow 2 by @tatiana in #2119

Enhancements

* Remove usage of contextmanager in plugins for accessing connections in
Airflow >= 3.1.2 by @pankajkoti in #2073

Docs

* Improve ``ExecutionMode.AIRFLOW_ASYNC`` docs by @tatiana in #2103
* Add note about experimenting threads count for the Watcher Execution
mode by @pankajkoti in #2083
* Fix minor documentation formatting issue by @dnskrv in #2098
* Correct example YAML key from ``operator_args`` to ``operator_kwargs``
by @jx2lee in #2091

Others

* Fix broken CI due to fastapi incompatibility with cadwyn for Airflow 3
by @pankajkoti in #2076
* pre-commit autoupdate in #2078, #2104

Closes: astronomer/oss-integrations-private#272
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test Airflow 3.1.1 connection access from Airflow 3 plugin and cleanup context manager usage

3 participants