Add dbt Fusion documentation#1824
Merged
Merged
Conversation
Related to changes introduced in PR #1803
✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.
|
Contributor
There was a problem hiding this comment.
Pull Request Overview
Adds initial documentation for the new dbt Fusion feature in Cosmos.
- Includes dbt Fusion in the configuration TOC.
- Provides a dedicated
dbt-fusion.rstwith context, setup, usage, and limitations.
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| docs/configuration/index.rst | Add “dbt Fusion” entry to the toctree |
| docs/configuration/dbt-fusion.rst | New documentation page for dbt Fusion |
Comments suppressed due to low confidence (2)
docs/configuration/index.rst:11
- The reference target
dbt-fusiondoes not match the anchordbt_fusiondefined in dbt-fusion.rst. Rename one to match the other so the link resolves correctly.
dbt Fusion <dbt-fusion>
docs/configuration/dbt-fusion.rst:33
- This file uses Markdown-style fences (```), which aren’t valid in reStructuredText. Use an RST literal block or
.. code-block:: bashdirective instead.
</details>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1824 +/- ##
=======================================
Coverage 98.01% 98.01%
=======================================
Files 85 85
Lines 5290 5290
=======================================
Hits 5185 5185
Misses 105 105 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
yetudada
reviewed
Jun 23, 2025
Contributor
yetudada
left a comment
yetudada
left a comment
There was a problem hiding this comment.
I left quick comments that you may need to check formatting on!
Co-authored-by: Yetunde Dada <43755008+yetudada@users.noreply.github.com>
Co-authored-by: Yetunde Dada <43755008+yetudada@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Collaborator
Author
|
Thank you very much, @yetudada and @pankajastro for the reviews! |
tatiana
added a commit
that referenced
this pull request
Jun 30, 2025
dbt Labs [launched](https://www.getdbt.com/blog/dbt-launch-showcase-2025-recap) [dbt Fusion](https://github.com/dbt-labs/dbt-fusion) on 28 May 2025. It is a unified approach that aims to merge dbt Core and dbt Cloud features via a completely new CLI, implemented in a different programming language (Rust, as opposed to Python). As part of this, dbt Labs are [rewriting all dbt adaptors](https://github.com/dbt-labs/dbt-fusion/tree/main/crates/dbt-fusion-adapter/src/adapters) (equivalent to Airflow providers) in Rust, starting from Snowflake. They are also changing the licensing model to a hybrid Open-Source and commercial license, which has implications for Cosmos. Among the advantages of using dbtFusion is a significant performance improvement (claimed to be 30x, for some models) and AI features. This PR adds initial support for running dbt Fusion with Cosmos when using `ExectionMode.LOCAL`. We do not have a solution for using `ExecutionMode.AIRFLOW_ASYNC` (https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes.html#airflow-async). Since dbt Fusion exposes itself both as dbt (like dbt-core) and optionally as dbtf, so there’s no need to change the default binary references in Cosmos. The documentation for this feature is being proposed in a separate PR: #1824 # How to use 1. Install dbt Fusion in your Airflow deployment End-users should install the dbt Fusion package themselves. An example of how to do this in Astro would be to add the following lines in your `Dockerfile`: ``` USER root RUN apt install -y curl RUN curl -fsSL https://public.cdn.getdbt.com/fs/install/install.sh | sh -s -- --update ``` 2. Update your `DbtDag` or `DbtTaskGroup` to use the dbt Fusion binary Example: ``` DbtDag( ..., execution_config=ExecutionConfig( dbt_executable_path="/home/astro/.local/bin/dbt" ) ) ``` Closes: #1792
Closed
Merged
tatiana
added a commit
that referenced
this pull request
Oct 29, 2025
**Features** * Introduce ``ExecutionMode.WATCHER`` to reduce DAG run time by 1/5 in several PRs. Learn more about it [here](https://astronomer.github.io/astronomer-cosmos/getting_started/watcher-execution-mode.html#watcher-execution-mode). This feature was implemented via multiple PRs, including: * Expose new execution mode by @tatiana @pankajastro @pankajkoti in #1999 * Add ``DbtProducerWatcherOperator`` for the proposed ``ExecutionMode.WATCHER`` by @pankajkoti in #1982 * Add ``DbtConsumerWatcherSensor`` for the proposed ``ExecutionMode.WATCHER`` by @pankajastro in #1998 * Push producer's task completion status to XCOM by @pankajkoti in #2000 * Add default priority_weight for ``DbtProducerWatcherOperator`` by @pankajkoti in #1995 * Add sample dbt events for the dbt watcher execution mode by @pankajkoti in #1952 * Add ``compiled_sql`` as a template fields on ```ExecutionMode.WATCHER``` when using ``run_results.json`` by @pankajastro in #2070 * Set ``push_run_results_to_xcom`` kwargs correctly for invocation mode subprocess and Watcher mode by @pankajastro in #2067 * Store compiled SQL as template field for dbt callback events in ``ExecutionMode.WATCHER`` by @pankajkoti in #2068 * Add initial documentation for ``ExecutionMode.WATCHER`` by @tatiana in #2046 * Support running ``State.UPSTREAM_FAILED`` tasks when WATCHER consumer upstream tasks fail by @tatiana in #2062 * Fail sensor tasks immediately if the ``ExecutionMode.WATCHER`` producer task fails by @pankajastro in #2040 * Add ``WATCHER``` to GitHub issue template by @tatiana in #2056 * Add support for ``TestBehavior.AFTER_ALL`` with ``ExecutionMode.WATCHER`` by @pankajastro in #2049 * Add support for ``TestBehavior.NONE`` with ``ExecutionMode.WATCHER`` by @pankajastro in #2047 * Fix ``ExecutionMode.WATCHER`` behaviour with ``DbtTaskGroup`` by @tatiana in #2044 * Fix Cosmos behaviour when using watcher with ``InvocationMode.DBT_RUNNER`` by @tatiana in #2048 * Add Airflow 3 plugin for dbt docs with multiple dbt projects support by @pankajkoti in #2009, check the [documentation](https://astronomer.github.io/astronomer-cosmos/configuration/hosting-docs.html). * Initial support to ``dbt Fusion`` by @tatiana in #1803. More details [here](https://astronomer.github.io/astronomer-cosmos/configuration/dbt-fusion). * Support to prune sources without downstream references in dbt projects by @corsettigyg in #1988 * Allow to set task display name as a user-defined function by @corsettigyg in #1761 * Add dbt project's hash to dag docs to support dag versioning in Airflow 3 by @pankajkoti in #1907 * feat: Add Jinja templating support for ``dbt_cmd_flags`` by @skillicinski in #1899 * Add Scarf metric to collect the execution mode uses by @pankajastro in #1981 * Support Airflow 3.1 by @tatiana in #1980 * Add MySQL profile mapping by @Lee2532 in #1977 * Add sqlserver profile mapping by @pankajastro in #1737 **Enhancement** * Use XCom to store sql when using ``ExecutionMode.AIRFLOW_ASYNC`` by @pankajastro in #1934 * Refactor ``AIRFLOW_ASYNC`` teardown so it doesn't install the virtualenv by @pankajastro in #1938 * Reuse the virtual env for ``AIRFLOW_ASYNC`` setup task by @pankajastro in #1939 * Improve dataset/asset experience in Cosmos by @tatiana in #2030 * Add ``downstreams`` to ``DbtNode`` by @wornjs in #2028 **Bug fixes** * Fix tags extraction by @ms32035 in #1915 * Fix task flow operator args by @anyapriya in #2024 **Documentation** * Add documentation for Airflow 3 Plugin supporting dbt docs for multiple dbt projects by @pankajkoti in #2063 * Add Cosmos Deferrable Operator Guide by @pankajastro in #1922 * Add dbt Fusion documentation by @tatiana in #1824 #1830 * Update dbt-fusion.rst to explicitly highlight it is in alpha by @tatiana in #1838 * Fix a bunch of docs build errors and warnings by @pankajkoti in #1886 * Add docs note for param virtualenv_dir for async execution mode by @pankajastro in #1969 * Use pepy.tech downloads badge in README by @pankajkoti in #1920 * Correct the default value of ``cache_dir`` by @seokyun.ha in #2027 **Others** * Promote @corsettigyg to committer by @tatiana in #1985 * Add @pankajkoti and @pankajastro to ``contributors.rst`` by @tatiana in #1983 * Update setup script for airflow3 script by @dwreeves in #2023 * Prevent pytest from trying to test classes that aren't actually tests by @anyapriya in #2032 * Fix ``dag.test()`` for Airflow 3.1+ by syncing DAG to database bby @kaxil in #2037 * Disable Scarf in CI by @pankajastro in #2016 * Fix failing dbt Fusion tests when run in parallel in CI by @pankajkoti in #1896 * Fix MyPy issues related to ``ObjectStoragePath`` in main branch by @tatiana in #2012 * Cleanup example dbt event JSON dictionaries kept for XCOM referencby @pankajkoti in #1997 * Bump min hatch version that includes fixes for click>=8.3.0 by @pankajkoti in #1996 * Use official postgres image from Docker hub for kubernetes setup by @pankajkoti in #1986 * Use click<8.3.0 for hatch as click 8.3 breaks hatch by @pankajkoti in #1987 * Pin Airflow version in type check CI job by @pankajastro in #2003 * Improve comments after feedback on #1948 by @tatiana in #1963 * Fix running tests with dbt Fusion 2.0.0 preview versions by @tatiana in #1948 * Test hardening of dbt node having tags as unset or missing by @pankajkoti in #1918 * Fix Sphinx issue in the main branch by @tatiana in #2064 * pre-commit autoupdate in #2065, #2043, #2033, #2019, #1990, #2019, #2008, #1941, #1935, #1924 * GitHub dependabot update in #2051, #2050, #2038, #2022, #1947, #1955, #1946, #1944, #1945, #1928, #1921, #1917 Co-authored-by: Pankaj Koti <pankaj.koti@astronomer.io> Co-authored-by: Pankaj Singh <pankaj.singh@astronomer.io> Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add documentation to Cosmos support for dbt Fusion, introduced in PR #1803