diff --git a/docs/configuration/cosmos-conf.rst b/docs/configuration/cosmos-conf.rst index 7ff0ddfa3a..a501c72bae 100644 --- a/docs/configuration/cosmos-conf.rst +++ b/docs/configuration/cosmos-conf.rst @@ -96,6 +96,13 @@ This page lists all available Airflow configurations that affect ``astronomer-co - Default: ``True`` - Environment Variable: ``AIRFLOW__COSMOS__ENABLE_CACHE_PROFILE`` +.. _pre_dbt_fusion: + From Cosmos 1.11, we have introduced support for dbt Fusion. Some of the changes may not be compatible with legacy versions of dbt-core. + If you find any issues on how Cosmos interacts with older versions of dbt-core you can use this configuration. + + - Default: ``False`` + - Environment Variable: ``AIRFLOW__COSMOS__PRE_DBT_FUSION`` + .. _profile_cache_dir_name: `profile_cache_dir_name`_: diff --git a/docs/configuration/dbt-fusion.rst b/docs/configuration/dbt-fusion.rst new file mode 100644 index 0000000000..d119b0adb3 --- /dev/null +++ b/docs/configuration/dbt-fusion.rst @@ -0,0 +1,70 @@ +.. _dbt_fusion: + +dbt Fusion support +================== + +.. note:: + Available since Cosmos 1.11 when using ``ExecutionMode.LOCAL``. + +Context +------- + +dbt Labs `launched `_ `dbt Fusion `_ on 28 May 2025. dbt Fusion is the next-generation dbt engine that enables real-time model validation, improved SQL parsing and state-aware orchestration. +It is a unified approach that aims to merge dbt Core and dbt Cloud features via a completely new CLI, +implemented in a different programming language (Rust, as opposed to Python). +As part of this, dbt Labs are `rewriting all dbt adapters `_ (equivalent to Airflow providers) in Rust, starting from Snowflake. +They are also changing the `licensing model `_ to a hybrid Open-Source and commercial license. We're supporting dbt Fusion with a license-complaint integration. This integration enables teams to: + - Use dbt Fusion locally for enhanced development experience with real-time validation + - Deploy dbt Fusion on Astro for production compliance with ELv2 licensing restrictions + - Maintain consistent workflows across development and production environments + +Some reported dbt Fusion features include: + - **Lightning-fast performance:** Up to 30× faster parsing speeds + - **Smarter orchestration:** Build only what’s changed, thanks to Fusion’s state-awareness + - **Real-time dev experience:** Catch errors instantly and explore lineage as you code, if you're using the VS Code extension + - **Multi-dialect support:** Instant error detection across Snowflake, Databricks, BigQuery, and Redshift SQL dialects + +> Note: dbt Fusion is in public beta with current support for Snowflake and Databricks projects, with additional data platforms coming soon. + +Support +------- + +Cosmos 1.11 adds initial support to running dbt Fusion with Cosmos when using ``ExecutionMode.LOCAL``. + +We do not have a solution for using `ExecutionMode.AIRFLOW_ASYNC `_. + +How to use +---------- + +1. Install dbt Fusion in your Airflow deployment + +End-users should install the dbt Fusion package themselves. An example of how to do this in Astro would be to add the following lines in your ``Dockerfile``: + +``` +USER root +RUN apt install -y curl +RUN curl -fsSL https://public.cdn.getdbt.com/fs/install/install.sh | sh -s -- --update +``` + +2. Update your ``DbtDag`` or ``DbtTaskGroup`` to use the dbt Fusion binary + +Example: + +``` +DbtDag( + ..., + execution_config=ExecutionConfig( + dbt_executable_path="/home/astro/.local/bin/dbt" + ) +) +``` + +Limitations +----------- + +- Currently (23 June 2025) dbt Fusion is still in beta +- dbt Fusion only supports Snowflake +- Cosmos does not support dbt Fusion when using ``ExecutionMode.AIRFLOW_ASYNC`` +- To support dbt Fusion, Cosmos changed how it interacts with dbt. This works with the latest versions of dbt-core, but may not +work with older versions. If you want to continue using dbt-core and was affected, set the environment variable +``AIRFLOW__COSMOS__PRE_DBT_FUSION=1`` and Cosmos interaction with dbt-core will work as previous versions. diff --git a/docs/configuration/index.rst b/docs/configuration/index.rst index 55ba51787f..8ef3e3f533 100644 --- a/docs/configuration/index.rst +++ b/docs/configuration/index.rst @@ -8,6 +8,8 @@ Cosmos offers a number of configuration options to customize its behavior. For m .. toctree:: :caption: Contents: + dbt Fusion + Project Config Profile Config Execution Config