Skip to content

Support Airflow 3.1#1980

Merged
tatiana merged 92 commits into
mainfrom
af-31
Oct 22, 2025
Merged

Support Airflow 3.1#1980
tatiana merged 92 commits into
mainfrom
af-31

Conversation

@tatiana
Copy link
Copy Markdown
Collaborator

@tatiana tatiana commented Sep 16, 2025

Apache Airflow 3.1 beta2 was released on 15 September 2025. We have been working on supporting Airflow 3.1 since its initial release on September 25, 2025.

These were some of the issues we faced while attempting to support Airflow 3.1, seen on the tests run in:
https://github.com/astronomer/astronomer-cosmos/actions/runs/17766456237/job/50490975762

Example of unittests that fail when running:

hatch run tests.py3.10-3.1.0b2-1.10:test-cov

Errors:

FAILED tests/dbt/test_graph.py::test_save_dbt_ls_cache_remote_cache_dir - AttributeError: The module `'airflow.io'` has no attribute `'path'`
FAILED tests/dbt/test_graph.py::test_get_dbt_ls_cache_remote_cache_dir - AttributeError: The module `'airflow.io'` has no attribute `'path'`
FAILED tests/operators/_asynchronous/test_base.py::test_execute_removes_existing_path - AttributeError: The module `'airflow.io'` has no attribute `'path'`
FAILED tests/operators/test_kubernetes.py::test_dbt_kubernetes_operator_handle_warnings[\n        19:48:25  Concurrency: 4 threads (target='target')\n        19:48:25\n        19:48:25  1 of 2 START test dbt_utils_accepted_range_table_col__12__0 ................... [RUN]\n        19:48:25  2 of 2 START test unique_table__uuid .......................................... [RUN]\n        19:48:27  1 of 2 WARN dbt_utils_accepted_range_table_col__12__0 ..................... [WARN in 1.83s]\n        19:48:27  2 of 2 PASS unique_table__uuid ................................................ [PASS in 1.85s]\n        19:48:27\n        19:48:27  Finished running 2 tests, 1 hook in 0 hours 0 minutes and 12.86 seconds (12.86s).\n        19:48:27\n        19:48:27  Completed with 1 warning:\n        19:48:27\n        19:48:27  Warning in test dbt_utils_accepted_range_table_col__12__0 (models/ads/ads.yaml)\n        19:48:27  Got 252 results, configured to warn if >0\n        19:48:27\n        19:48:27    compiled Code at target/compiled/model/models/table/table.yaml/dbt_utils_accepted_range_table_col__12__0.sql\n        19:48:27\n        19:48:27  Done. PASS=1 WARN=1 ERROR=0 SKIP=0 TOTAL=2\n        19:48:27  Command `dbt test` succeeded at 07:50:02.340364 after 43.98 seconds\n        19:48:27  Flushing usage events\n        -True] - TypeError: __init__() missing 1 required positional argument: 'dag_version_id'
FAILED tests/operators/test_kubernetes.py::test_dbt_kubernetes_operator_handle_warnings[\n        19:48:25  Concurrency: 4 threads (target='target')\n        19:48:25\n        19:48:25  1 of 2 START test dbt_utils_accepted_range_table_col__12__0 ................... [RUN]\n        19:48:25  2 of 2 START test unique_table__uuid .......................................... [RUN]\n        19:48:27  1 of 2 PASS 252 dbt_utils_accepted_range_table_col__12__0 ..................... [PASS in 1.83s]\n        19:48:27  2 of 2 PASS unique_table__uuid ................................................ [PASS in 1.85s]\n        19:48:27\n        19:48:27  Finished running 2 tests, 1 hook in 0 hours 0 minutes and 12.86 seconds (12.86s).\n        19:48:27\n        19:48:27  Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2\n        -False] - TypeError: __init__() missing 1 required positional argument: 'dag_version_id'
FAILED tests/operators/test_kubernetes.py::test_dbt_kubernetes_operator_handle_warnings[\n        gibberish\n        -False] - TypeError: __init__() missing 1 required positional argument: 'dag_version_id'
FAILED tests/operators/test_kubernetes.py::test_dbt_kubernetes_operator_handle_warnings_noop - TypeError: __init__() missing 1 required positional argument: 'dag_version_id'
FAILED tests/operators/test_local.py::test_run_operator_emits_events_without_openlineage_events_completes - TypeError: __init__() missing 1 required positional argument: 'dag_version_id'
FAILED tests/operators/test_local.py::test_configure_remote_target_path - AttributeError: The module `'airflow.io'` has no attribute `'path'`
FAILED tests/operators/test_local.py::test_upload_compiled_sql_should_upload - AttributeError: The module `'airflow.io'` has no attribute `'path'`
FAILED tests/operators/test_local.py::test_upload_sql_files_creates_parent_directories - AttributeError: The module `'airflow.io'` has no attribute `'path'`
FAILED tests/operators/test_virtualenv.py::test_run_command_without_virtualenv_dir - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/operators/test_virtualenv.py::test_run_command_with_virtualenv_dir - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/athena/test_athena_access_key.py::test_athena_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/bigquery/test_bq_service_account_file.py::test_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/clickhouse/test_clickhouse_userpass.py::test_connection_claiming1 - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/databricks/test_dbr_oauth.py::test_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/databricks/test_dbr_token.py::test_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/duckdb/test_duckdb_user_pass.py::test_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/exasol/test_exasol_user_pass.py::test_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/exasol/test_exasol_user_pass.py::test_dsn_formatting - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/oracle/test_oracle_user_pass.py::test_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/oracle/test_oracle_user_pass.py::test_invalid_connection_type - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/postgres/test_pg_user_pass.py::test_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/redshift/test_redshift_user_pass.py::test_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/snowflake/test_snowflake_user_encrypted_privatekey_env_variable.py::test_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/snowflake/test_snowflake_user_encrypted_privatekey_env_variable.py::test_old_snowflake_format - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/snowflake/test_snowflake_user_encrypted_privatekey_file.py::test_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/snowflake/test_snowflake_user_encrypted_privatekey_file.py::test_old_snowflake_format - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/snowflake/test_snowflake_user_pass.py::test_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/snowflake/test_snowflake_user_pass.py::test_old_snowflake_format - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/snowflake/test_snowflake_user_pass.py::test_appends_region - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/snowflake/test_snowflake_user_pass.py::test_appends_host_and_port - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/snowflake/test_snowflake_user_privatekey.py::test_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/snowflake/test_snowflake_user_privatekey.py::test_old_snowflake_format - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/snowflake/test_snowflake_user_privatekey.py::test_appends_region - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/spark/test_spark_thrift.py::test_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/sqlserver/test_standard_sqlserver_auth.py::test_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/teradata/test_teradata_user_pass.py::test_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/trino/test_trino_base.py::test_profile_args - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/trino/test_trino_base.py::test_profile_args_overrides - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/trino/test_trino_certificate.py::test_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
FAILED tests/profiles/trino/test_trino_jwt.py::test_connection_claiming - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/snowflake/test_snowflake_user_encrypted_privatekey_file.py::test_profile_mapping_selected - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/snowflake/test_snowflake_user_encrypted_privatekey_file.py::test_profile_args - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/snowflake/test_snowflake_user_encrypted_privatekey_file.py::test_profile_args_overrides - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/snowflake/test_snowflake_user_encrypted_privatekey_file.py::test_profile_env_vars - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/snowflake/test_snowflake_user_pass.py::test_profile_mapping_selected - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/snowflake/test_snowflake_user_pass.py::test_profile_args - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/snowflake/test_snowflake_user_pass.py::test_profile_args_overrides - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/snowflake/test_snowflake_user_pass.py::test_profile_env_vars - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/snowflake/test_snowflake_user_privatekey.py::test_decode_private_key_content[base64_encoded] - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/snowflake/test_snowflake_user_privatekey.py::test_decode_private_key_content[plain_text] - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/snowflake/test_snowflake_user_privatekey.py::test_profile_mapping_selected - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/snowflake/test_snowflake_user_privatekey.py::test_profile_args - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/snowflake/test_snowflake_user_privatekey.py::test_profile_args_overrides - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/snowflake/test_snowflake_user_privatekey.py::test_profile_env_vars - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/snowflake/test_snowflake_user_privatekey.py::test_profile_env_vars_with_base64 - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/spark/test_spark_thrift.py::test_spark_mapping_selected - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/spark/test_spark_thrift.py::test_profile_args - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/spark/test_spark_thrift.py::test_profile_args_overrides - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/spark/test_spark_thrift.py::test_profile_env_vars - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/sqlserver/test_standard_sqlserver_auth.py::test_profile_mapping_selected - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/sqlserver/test_standard_sqlserver_auth.py::test_profile_args - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/sqlserver/test_standard_sqlserver_auth.py::test_profile_env_vars - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/teradata/test_teradata_user_pass.py::test_profile_mapping_selected - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/teradata/test_teradata_user_pass.py::test_profile_mapping_schema_validation - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/teradata/test_teradata_user_pass.py::test_profile_mapping_keeps_port - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/teradata/test_teradata_user_pass.py::test_profile_mapping_keeps_custom_tmode - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/teradata/test_teradata_user_pass.py::test_profile_args - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/teradata/test_teradata_user_pass.py::test_profile_args_overrides - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/teradata/test_teradata_user_pass.py::test_profile_env_vars - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/trino/test_trino_certificate.py::test_trino_mapping_selected - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/trino/test_trino_certificate.py::test_profile_args - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/trino/test_trino_certificate.py::test_profile_args_overrides - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/trino/test_trino_jwt.py::test_trino_mapping_selected - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/trino/test_trino_jwt.py::test_profile_args - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/trino/test_trino_jwt.py::test_profile_args_overrides - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/trino/test_trino_jwt.py::test_profile_env_vars - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/trino/test_trino_ldap.py::test_trino_mapping_selected - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/trino/test_trino_ldap.py::test_profile_args - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/trino/test_trino_ldap.py::test_profile_args_overrides - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/trino/test_trino_ldap.py::test_profile_env_vars - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/vertica/test_vertica_user_pass.py::test_profile_mapping_selected - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/vertica/test_vertica_user_pass.py::test_profile_mapping_keeps_custom_port - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/vertica/test_vertica_user_pass.py::test_profile_args - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/vertica/test_vertica_user_pass.py::test_profile_args_overrides - AttributeError: module 'airflow.hooks' has no attribute 'base'
ERROR tests/profiles/vertica/test_vertica_user_pass.py::test_profile_env_vars - AttributeError: module 'airflow.hooks' has no attribute 'base'
= 50 failed, 660 passed, 5 skipped, 75 deselected, 2 xfailed, 6 warnings, 138 errors in 63.25s (0:01:03) =

I created a follow-up ticket for the last three tests that I didn't fix in Airflow 3.1:
#2045

Closes: https://github.com/astronomer/oss-integrations-private/issues/209

@netlify
Copy link
Copy Markdown

netlify Bot commented Sep 16, 2025

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit 0a9fdbb
🔍 Latest deploy log https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/68f20d545ba716000830b807

Comment thread .github/workflows/test.yml Outdated
apache-airflow==3.1.0b2 depends on Python>=3.10,<3.14
@codecov
Copy link
Copy Markdown

codecov Bot commented Sep 16, 2025

Codecov Report

❌ Patch coverage is 92.74194% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.71%. Comparing base (4ccf9f7) to head (322e660).

Files with missing lines Patch % Lines
cosmos/operators/_asynchronous/bigquery.py 0.00% 4 Missing ⚠️
cosmos/cache.py 71.42% 2 Missing ⚠️
cosmos/io.py 85.71% 2 Missing ⚠️
cosmos/operators/local.py 97.56% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1980      +/-   ##
==========================================
- Coverage   97.81%   97.71%   -0.11%     
==========================================
  Files          89       89              
  Lines        5623     5681      +58     
==========================================
+ Hits         5500     5551      +51     
- Misses        123      130       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tatiana tatiana changed the title [WIP] Run tests against Airflow 3.1 [WIP] Support Airflow 3.1 Sep 16, 2025
Comment thread tests/listeners/test_dag_run_listener.py
Comment thread dev/dags/example_watcher.py Outdated
FAILED tests/operators/test_virtualenv.py::test_integration_virtualenv_operator - AttributeError: 'NoneType' object has no attribute 'state'
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for Apache Airflow 3.1 to Cosmos by addressing breaking API changes and deprecated modules. The primary changes involve adapting to new import paths for BaseHook, ObjectStoragePath, TaskGroup, and handling test framework changes for TaskInstance initialization.

Key changes:

  • Updated import paths for modules relocated in Airflow 3.1 (BaseHook, ObjectStoragePath, TaskGroup)
  • Fixed test compatibility by adapting to new TaskInstance signature requirements
  • Updated CI/CD configurations to include Airflow 3.1 in the test matrix

Reviewed Changes

Copilot reviewed 76 out of 76 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
cosmos/profiles/base.py Added fallback imports for BaseHook moved to airflow.sdk.bases.hook
cosmos/config.py Updated ObjectStoragePath imports to support airflow.sdk location
cosmos/operators/local.py Consolidated ObjectStoragePath imports and replaced self.log with logger
cosmos/operators/watcher.py Updated logging patterns and template_fields configurations
tests/* (multiple files) Updated mock patch paths from airflow.hooks.base to cosmos.profiles.base
tests/operators/test_*.py Added conditional TaskInstance initialization for Airflow 3.1 compatibility
pyproject.toml Added Airflow 3.1 to test matrix
.github/workflows/test.yml Updated CI to test Airflow 3.1

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread cosmos/operators/local.py Outdated
Comment thread tests/test_io.py
Comment thread scripts/test/pre-install-airflow.sh Outdated
Comment thread scripts/test/pre-install-airflow.sh Outdated
Comment thread tests/listeners/test_dag_run_listener.py
Comment thread scripts/test/pre-install-airflow.sh Outdated
Comment thread cosmos/operators/_asynchronous/bigquery.py Outdated
Copy link
Copy Markdown
Contributor

@pankajkoti pankajkoti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great persistence! 👏🏽

Comment thread cosmos/operators/local.py Outdated
Comment thread cosmos/operators/watcher.py Outdated
Comment thread cosmos/io.py Outdated
Comment thread scripts/test/pre-install-airflow.sh Outdated
Comment thread tests/operators/test_local.py Outdated
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Comment thread cosmos/io.py Outdated
Comment thread cosmos/operators/watcher.py Outdated
tatiana and others added 2 commits October 22, 2025 09:23
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Comment thread scripts/test/pre-install-airflow.sh Outdated
Comment thread scripts/test/pre-install-airflow.sh Outdated
tatiana and others added 2 commits October 22, 2025 09:29
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@tatiana tatiana mentioned this pull request Oct 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants