Skip to content

Add experimental support to dbt Fusion#1803

Merged
tatiana merged 25 commits into
mainfrom
dbtf
Jun 30, 2025
Merged

Add experimental support to dbt Fusion#1803
tatiana merged 25 commits into
mainfrom
dbtf

Conversation

@tatiana
Copy link
Copy Markdown
Collaborator

@tatiana tatiana commented Jun 10, 2025

Context

dbt Labs launched dbt Fusion on 28 May 2025. It is a unified approach that aims to merge dbt Core and dbt Cloud features
via a completely new CLI, implemented in a different programming
language (Rust, as opposed to Python). As part of this, dbt Labs are
rewriting all dbt
adaptors

(equivalent to Airflow providers) in Rust, starting from Snowflake. They
are also changing the licensing model to a hybrid Open-Source and
commercial license, which has implications for Cosmos. Among the
advantages of using dbtFusion is a significant performance improvement
(claimed to be 30x, for some models) and AI features.

This PR adds initial support for running dbt Fusion with Cosmos when
using ExectionMode.LOCAL.

We do not have a solution for using ExecutionMode.AIRFLOW_ASYNC
(https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes.html#airflow-async).

Since dbt Fusion exposes itself both as dbt (like dbt-core) and
optionally as dbtf, so there’s no need to change the default binary references in Cosmos.

The documentation for this feature is being proposed in a separate PR:
#1824

How to use

  1. Install dbt Fusion in your Airflow deployment

End-users should install the dbt Fusion package themselves. An example
of how to do this in Astro would be to add the following lines in your
Dockerfile:

USER root
RUN apt install -y curl
RUN curl -fsSL https://public.cdn.getdbt.com/fs/install/install.sh | sh -s -- --update
  1. Update your DbtDag or DbtTaskGroup to use the dbt Fusion binary

Example:

DbtDag(
    ...,
    execution_config=ExecutionConfig(
        dbt_executable_path="/home/astro/.local/bin/dbt"
    )
)

Closes: #1792

@netlify
Copy link
Copy Markdown

netlify Bot commented Jun 10, 2025

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit 1d25b8e
🔍 Latest deploy log https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/686279d6a988880008e98b98

@cloudflare-workers-and-pages
Copy link
Copy Markdown

Deploying astronomer-cosmos with  Cloudflare Pages  Cloudflare Pages

Latest commit: 4f6f85b
Status: ✅  Deploy successful!
Preview URL: https://aa4b467d.astronomer-cosmos.pages.dev
Branch Preview URL: https://dbtf.astronomer-cosmos.pages.dev

View logs

tatiana and others added 7 commits June 20, 2025 15:41
dbtf run --models customers --profile snowflake
error: unexpected argument '--models' found
[2025-06-02T07:20:08.734+0000] {graph.py:317} INFO - Could not parse following the dbt ls line even though it was a valid JSON `{"name":"not_null_stg_customers_customer_id","package_name":"jaffle_shop","path":"generic_tests/not_null_stg_customers_customer_id.sql","resource_type":"test","unique_id":"test.jaffle_shop.not_null_stg_customers_customer_id"}`
It was not displaying the depends_on relationship
@tatiana
Copy link
Copy Markdown
Collaborator Author

tatiana commented Jun 27, 2025

Thank you a lot for the feedback, @RRDTIL. I'm glad to hear it worked out.

Please, could you confirm if there are still any issues with the docs?

Please, do let us know if you have any feedback on 1.11.0a1!

: Unknown pytest.mark.dbtFusion - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.dbtFusion
@tatiana tatiana merged commit e836125 into main Jun 30, 2025
167 checks passed
@tatiana tatiana deleted the dbtf branch June 30, 2025 12:38
@RRDTIL
Copy link
Copy Markdown

RRDTIL commented Jun 30, 2025

@tatiana

Please, could you confirm if there are still any issues with the docs?

Problem solved, the documentation renders perfectly !

Please, do let us know if you have any feedback on 1.11.0a1!

I'll do it as soon as I can, but at the speed you're going, I won't have time to test the pre-release before you've already published a release 😆

@tatiana tatiana mentioned this pull request Aug 8, 2025
@tatiana tatiana mentioned this pull request Oct 7, 2025
@tatiana tatiana added this to the Cosmos 1.11.0 milestone Oct 28, 2025
tatiana added a commit that referenced this pull request Oct 29, 2025
**Features**

* Introduce ``ExecutionMode.WATCHER`` to reduce DAG run time by 1/5 in
several PRs. Learn more about it
[here](https://astronomer.github.io/astronomer-cosmos/getting_started/watcher-execution-mode.html#watcher-execution-mode).
This feature was implemented via multiple PRs, including:
* Expose new execution mode by @tatiana @pankajastro @pankajkoti in
#1999
* Add ``DbtProducerWatcherOperator`` for the proposed
``ExecutionMode.WATCHER`` by @pankajkoti in #1982
* Add ``DbtConsumerWatcherSensor`` for the proposed
``ExecutionMode.WATCHER`` by @pankajastro in #1998
* Push producer's task completion status to XCOM by @pankajkoti in #2000
* Add default priority_weight for ``DbtProducerWatcherOperator`` by
@pankajkoti in #1995
* Add sample dbt events for the dbt watcher execution mode by
@pankajkoti in #1952
* Add ``compiled_sql`` as a template fields on
```ExecutionMode.WATCHER``` when using ``run_results.json`` by
@pankajastro in #2070
* Set ``push_run_results_to_xcom`` kwargs correctly for invocation mode
subprocess and Watcher mode by @pankajastro in #2067
* Store compiled SQL as template field for dbt callback events in
``ExecutionMode.WATCHER`` by @pankajkoti in #2068
* Add initial documentation for ``ExecutionMode.WATCHER`` by @tatiana in
#2046
* Support running ``State.UPSTREAM_FAILED`` tasks when WATCHER consumer
upstream tasks fail by @tatiana in #2062
* Fail sensor tasks immediately if the ``ExecutionMode.WATCHER``
producer task fails by @pankajastro in #2040
  * Add ``WATCHER``` to GitHub issue template by @tatiana in #2056
* Add support for ``TestBehavior.AFTER_ALL`` with
``ExecutionMode.WATCHER`` by @pankajastro in #2049
* Add support for ``TestBehavior.NONE`` with ``ExecutionMode.WATCHER``
by @pankajastro in #2047
* Fix ``ExecutionMode.WATCHER`` behaviour with ``DbtTaskGroup`` by
@tatiana in #2044
* Fix Cosmos behaviour when using watcher with
``InvocationMode.DBT_RUNNER`` by @tatiana in #2048

* Add Airflow 3 plugin for dbt docs with multiple dbt projects support
by @pankajkoti in #2009, check the
[documentation](https://astronomer.github.io/astronomer-cosmos/configuration/hosting-docs.html).
* Initial support to ``dbt Fusion`` by @tatiana in #1803. More details
[here](https://astronomer.github.io/astronomer-cosmos/configuration/dbt-fusion).
* Support to prune sources without downstream references in dbt projects
by @corsettigyg in #1988
* Allow to set task display name as a user-defined function by
@corsettigyg in #1761
* Add dbt project's hash to dag docs to support dag versioning in
Airflow 3 by @pankajkoti in #1907
* feat: Add Jinja templating support for ``dbt_cmd_flags`` by
@skillicinski in #1899
* Add Scarf metric to collect the execution mode uses by @pankajastro in
#1981
* Support Airflow 3.1 by @tatiana in #1980
* Add MySQL profile mapping by @Lee2532 in #1977
* Add sqlserver profile mapping by @pankajastro in #1737

**Enhancement**

* Use XCom to store sql when using ``ExecutionMode.AIRFLOW_ASYNC`` by
@pankajastro in #1934
* Refactor ``AIRFLOW_ASYNC`` teardown so it doesn't install the
virtualenv by @pankajastro in #1938
* Reuse the virtual env for ``AIRFLOW_ASYNC`` setup task by @pankajastro
in #1939
* Improve dataset/asset experience in Cosmos by @tatiana in #2030
* Add ``downstreams`` to ``DbtNode`` by @wornjs in #2028

**Bug fixes**

* Fix tags extraction by @ms32035 in #1915
* Fix task flow operator args by @anyapriya in #2024

**Documentation**

* Add documentation for Airflow 3 Plugin supporting dbt docs for
multiple dbt projects by @pankajkoti in #2063
* Add Cosmos Deferrable Operator Guide by @pankajastro in #1922
* Add dbt Fusion documentation by @tatiana in #1824 #1830
* Update dbt-fusion.rst to explicitly highlight it is in alpha by
@tatiana in #1838
* Fix a bunch of docs build errors and warnings by @pankajkoti in
#1886
* Add docs note for param virtualenv_dir for async execution mode by
@pankajastro in #1969
* Use pepy.tech downloads badge in README by @pankajkoti in #1920
* Correct the default value of ``cache_dir`` by @seokyun.ha in #2027

**Others**

* Promote @corsettigyg to committer by @tatiana in #1985
* Add @pankajkoti and @pankajastro to ``contributors.rst`` by @tatiana
in #1983
* Update setup script for airflow3 script by @dwreeves in #2023
* Prevent pytest from trying to test classes that aren't actually tests
by @anyapriya in #2032
* Fix ``dag.test()`` for Airflow 3.1+ by syncing DAG to database bby
@kaxil in #2037
* Disable Scarf in CI by @pankajastro in #2016
* Fix failing dbt Fusion tests when run in parallel in CI by @pankajkoti
in #1896
* Fix MyPy issues related to ``ObjectStoragePath`` in main branch by
@tatiana in #2012
* Cleanup example dbt event JSON dictionaries kept for XCOM referencby
@pankajkoti in #1997
* Bump min hatch version that includes fixes for click>=8.3.0 by
@pankajkoti in #1996
* Use official postgres image from Docker hub for kubernetes setup by
@pankajkoti in #1986
* Use click<8.3.0 for hatch as click 8.3 breaks hatch by @pankajkoti in
#1987
* Pin Airflow version in type check CI job by @pankajastro in #2003
* Improve comments after feedback on #1948 by @tatiana in #1963
* Fix running tests with dbt Fusion 2.0.0 preview versions by @tatiana
in #1948
* Test hardening of dbt node having tags as unset or missing by
@pankajkoti in #1918
* Fix Sphinx issue in the main branch by @tatiana in #2064
* pre-commit autoupdate in #2065, #2043, #2033, #2019, #1990, #2019,
#2008, #1941, #1935, #1924
* GitHub dependabot update in #2051, #2050, #2038, #2022, #1947, #1955,
#1946, #1944, #1945, #1928, #1921, #1917


Co-authored-by: Pankaj Koti <pankaj.koti@astronomer.io>
Co-authored-by: Pankaj Singh <pankaj.singh@astronomer.io>
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] dbt-fusion compatibility

4 participants