diff --git a/docs/airflow3_compatibility/index.rst b/docs/airflow3_compatibility/index.rst index f0e5d0e0af..5668e92939 100644 --- a/docs/airflow3_compatibility/index.rst +++ b/docs/airflow3_compatibility/index.rst @@ -4,6 +4,32 @@ Airflow 3 Compatibility (First Iteration) The Cosmos 1.10.0 release marks the **first iteration** of adding compatibility for `Apache Airflow® 3 `_ This is an important milestone as we work towards ensuring that Cosmos seamlessly integrates with the latest advancements in the Airflow ecosystem. +Breaking changes +---------------- + +Airflow Asset (Dataset) URIs validation rules changed in Airflow 3.0.0 and OpenLineage URIs (standard used by Cosmos) are no longer valid in Airflow 3. + +Therefore, if using Cosmos with Airflow 3, the Airflow Dataset URIs will be changed to use backslashes instead of dots to separate the schema and table name. + +Example of Airflow 2 Cosmos Dataset URI: + +- postgres://0.0.0.0:5434/postgres.public.orders + +Example of Airflow 3 Cosmos Asset URI: + +- postgres://0.0.0.0:5434/postgres/public/orders + + +If you want to use the Airflow 3 URI standard while still using Airflow 2, please set: + +``` +export AIRFLOW__COSMOS__USE_DATASET_AIRFLOW3_URI_STANDARD=1 +``` + +.. warning:: + Remember to update any DAGs that are triggered using Cosmos-generated datasets or aliases to the new URI format. + + What Works ---------- diff --git a/docs/configuration/cosmos-conf.rst b/docs/configuration/cosmos-conf.rst index b055813f83..a3dcff0e02 100644 --- a/docs/configuration/cosmos-conf.rst +++ b/docs/configuration/cosmos-conf.rst @@ -178,6 +178,12 @@ This page lists all available Airflow configurations that affect ``astronomer-co - Default: ``True`` - Environment Variable: ``AIRFLOW__COSMOS__ENABLE_TEARDOWN_ASYNC_TASK`` +`use_dataset_airflow3_uri_standard`_: + (Introduced in Cosmos 1.10.0): Changes Cosmos Dataset (Asset) URIs to be Airflow 3 compliant. Since this would be a breaking change, it is False by default in Cosmos 1.x. + - Default: ``False`` + - Environment Variable: ``AIRFLOW__COSMOS__USE_DATASET_AIRFLOW3_URI_STANDARD`` + + [openlineage] ~~~~~~~~~~~~~ diff --git a/docs/configuration/scheduling.rst b/docs/configuration/scheduling.rst index 67ed1a10cb..da6ad9122b 100644 --- a/docs/configuration/scheduling.rst +++ b/docs/configuration/scheduling.rst @@ -132,6 +132,28 @@ From Cosmos 1.7 and Airflow 2.10, it is also possible to trigger DAGs be to be r Known Limitations ................. +Airflow 3.0 and beyond +______________________ + +Airflow Asset (Dataset) URIs validation rules changed in Airflow 3.0.0 and OpenLineage URIs (standard used by Cosmos) are no longer valid in Airflow 3. + +Therefore, if using Cosmos with Airflow 3, the Airflow Dataset URIs will be changed to use backslashes instead of dots to separate the schema and table name. + +Example of Airflow 2 Cosmos Dataset URI: +- postgres://0.0.0.0:5434/postgres.public.orders + +Example of Airflow 3 Cosmos Asset URI: +- postgres://0.0.0.0:5434/postgres/public/orders + + +If you want to use the Airflow 3 URI standard while still using Airflow 2, please set: + +``` +export AIRFLOW__COSMOS__USE_DATASET_AIRFLOW3_URI_STANDARD=1 +``` + +Remember to update any DAGs that are scheduled using this dataset. + Airflow 2.9 and below _____________________ @@ -175,3 +197,5 @@ they can set this configuration to ``False``. It can also be set in the ``airflo [cosmos] enable_dataset_alias = False + +Starting in Airflow 3, Cosmos users no longer allowed to set ``AIRFLOW__COSMOS__ENABLE_DATASET_ALIAS`` to ``True``.