Support running dbt deps incrementally to pre-defined dbt_packages during task execution

[tatiana]

Motivation

Support running dbt deps incrementally to pre-calculated dbt_packages during the task execution. This was a use case requested by an Astro customer.

Before this change, Cosmos supported two types of configuration:

• If users choose operator_args={"install_deps": False} or ProjectConfig.install_dbt_deps=False, Cosmos would create a symbolic link for the user's pre-defined dbt_packages (background: Refactor dbt ls to run from tempdir with symbolic links #488, Release 1.2.0 #600, Adding dbt_packages when dbt_deps is False #730)
• If users choose operator_args={"install_deps": True} or ProjectConfig.install_dbt_deps=True (default), Cosmos would ignore any user-predefined ' dbt_packagesand do a rundbt deps` from scratch from a temporary folder.

An Astronomer customer requested to reuse the defined initially dbt_packages directory and run dbt deps (incrementally).

Implementation

We do not run dbt commands directly in the original dbt project folder with Cosmos because some users use read-only filesystems (#414). We also decided to use symbolic links instead of copying the directory due to performance issues (#488). Since we did not want to introduce a breaking change in a minor Cosmos release by changing the existing Cosmos 1.x behaviour to meet this new use case, this PR supports:

• Copying the dbt deps related files (dbt packages folder and symbolic link) to the Cosmos temporary folder; and
• Running dbt deps.

So this is not a breaking change, users must opt into this behaviour by using ProjectConfig.copy_dbt_packages=True (new configuration) and operator_args={"install_dbt_deps": True} or ProjectConfig. install_dbt_deps =True and one of the following:

• Changing the operator to receive the argument copy_dbt_packages=True
• Changing individual DbtDag or DbtTaskGroup instances to use the new configuration ProjectConfig.copy_dbt_packages=True
• Changing the behaviour globally, by setting the Airflow configuration either via the environment variable AIRFLOW__COSMOS__DEFAULT_COPY_DBT_PACKAGES=True or via the airflow.cfg:

[cosmos]
default_copy_dbt_packages=True

How this was tested

To validate the end-to-end behaviour, we run the following dag from dev/dags:

airflow dags test dbt_deps_example

Related tickets

This is a follow-up to #1668 and #1669.

I'll make a follow-up PR covering the documentation.

Closes: #1630

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	43450bf
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/67ffc2c9a7d5d40008436027

[cloudflare-workers-and-pages]

Deploying astronomer-cosmos with    Cloudflare Pages

Latest commit:	43450bf
Status:	✅  Deploy successful!
Preview URL:	https://e3420e72.astronomer-cosmos.pages.dev
Branch Preview URL:	https://issue-1630-task-execution.astronomer-cosmos.pages.dev

View logs

[Copilot]

Pull Request Overview

This PR adds support for running dbt deps incrementally using a pre-defined dbt_packages directory by introducing a new configuration flag, copy_dbt_packages. Key changes include:

• Updating the operator to conditionally copy dbt_packages and adjust symbolic link creation based on the new configuration.
• Modifying the converter to pass the new copy_dbt_packages flag.
• Adjusting the dbt graph loading mode mapping for DBT_LS_CACHE.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
tests/operators/test_local.py	Added a test for _clone_project verifying symbolic link and copy paths.
cosmos/operators/local.py	Updated operator logic to support conditional copying and logging of dbt packages.
cosmos/dbt/graph.py	Changed mapping for DBT_LS_CACHE mode to use load_via_dbt_ls.
cosmos/converter.py	Added configuration override for the copy_dbt_packages flag.

Comments suppressed due to low confidence (2)

tests/operators/test_local.py:1531

• Consider adding a test case for _clone_project where copy_dbt_packages is false to ensure that symlink creation behaves as expected in that branch.

@patch("cosmos.operators.local.copy_dbt_packages")

cosmos/dbt/graph.py:531

• Mapping DBT_LS_CACHE to load_via_dbt_ls rather than load_via_dbt_ls_cache could lead to unintended behavior if the two loaders differ; double-check that this change is intentional and meets the expected functionality.

LoadMode.DBT_LS_CACHE: self.load_via_dbt_ls,

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.09%. Comparing base (5711580) to head (43450bf).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1670   +/-   ##
=======================================
  Coverage   97.08%   97.09%           
=======================================
  Files          80       80           
  Lines        5014     5022    +8     
=======================================
+ Hits         4868     4876    +8     
  Misses        146      146           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.