Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions docs/airflow3_compatibility/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,32 @@ Airflow 3 Compatibility (First Iteration)
The Cosmos 1.10.0 release marks the **first iteration** of adding compatibility for `Apache Airflow® 3 <https://airflow.apache.org/>`_
This is an important milestone as we work towards ensuring that Cosmos seamlessly integrates with the latest advancements in the Airflow ecosystem.

Breaking changes
----------------

Airflow Asset (Dataset) URIs validation rules changed in Airflow 3.0.0 and OpenLineage URIs (standard used by Cosmos) are no longer valid in Airflow 3.

Therefore, if using Cosmos with Airflow 3, the Airflow Dataset URIs will be changed to use backslashes instead of dots to separate the schema and table name.

Example of Airflow 2 Cosmos Dataset URI:

- postgres://0.0.0.0:5434/postgres.public.orders

Example of Airflow 3 Cosmos Asset URI:

- postgres://0.0.0.0:5434/postgres/public/orders


If you want to use the Airflow 3 URI standard while still using Airflow 2, please set:

```
export AIRFLOW__COSMOS__USE_DATASET_AIRFLOW3_URI_STANDARD=1
```

.. warning::
Remember to update any DAGs that are triggered using Cosmos-generated datasets or aliases to the new URI format.


What Works
----------

Expand Down
6 changes: 6 additions & 0 deletions docs/configuration/cosmos-conf.rst
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,12 @@ This page lists all available Airflow configurations that affect ``astronomer-co
- Default: ``True``
- Environment Variable: ``AIRFLOW__COSMOS__ENABLE_TEARDOWN_ASYNC_TASK``

`use_dataset_airflow3_uri_standard`_:
(Introduced in Cosmos 1.10.0): Changes Cosmos Dataset (Asset) URIs to be Airflow 3 compliant. Since this would be a breaking change, it is False by default in Cosmos 1.x.
- Default: ``False``
- Environment Variable: ``AIRFLOW__COSMOS__USE_DATASET_AIRFLOW3_URI_STANDARD``


[openlineage]
~~~~~~~~~~~~~

Expand Down
24 changes: 24 additions & 0 deletions docs/configuration/scheduling.rst
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,28 @@ From Cosmos 1.7 and Airflow 2.10, it is also possible to trigger DAGs be to be r
Known Limitations
.................

Airflow 3.0 and beyond
______________________

Airflow Asset (Dataset) URIs validation rules changed in Airflow 3.0.0 and OpenLineage URIs (standard used by Cosmos) are no longer valid in Airflow 3.

Therefore, if using Cosmos with Airflow 3, the Airflow Dataset URIs will be changed to use backslashes instead of dots to separate the schema and table name.

Example of Airflow 2 Cosmos Dataset URI:
- postgres://0.0.0.0:5434/postgres.public.orders

Example of Airflow 3 Cosmos Asset URI:
- postgres://0.0.0.0:5434/postgres/public/orders


If you want to use the Airflow 3 URI standard while still using Airflow 2, please set:
Comment thread
tatiana marked this conversation as resolved.

```
export AIRFLOW__COSMOS__USE_DATASET_AIRFLOW3_URI_STANDARD=1
```

Remember to update any DAGs that are scheduled using this dataset.

Airflow 2.9 and below
_____________________

Expand Down Expand Up @@ -175,3 +197,5 @@ they can set this configuration to ``False``. It can also be set in the ``airflo

[cosmos]
enable_dataset_alias = False

Starting in Airflow 3, Cosmos users no longer allowed to set ``AIRFLOW__COSMOS__ENABLE_DATASET_ALIAS`` to ``True``.
Loading