Fix ExecutionMode.LOCAL to leverage ProjectConfig.manifest_path

[tatiana]

Previously, Cosmos was ignoring the user-defined manifest file while executing tasks. This PR solves the problem.

Closes: #1643

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	3b6d504
🔍 Latest deploy log	https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/682ca2715cb02b0008f5f5a0

[cloudflare-workers-and-pages]

Deploying astronomer-cosmos with    Cloudflare Pages

Latest commit:	3b6d504
Status:	✅  Deploy successful!
Preview URL:	https://0c600896.astronomer-cosmos.pages.dev
Branch Preview URL:	https://fix-1643.astronomer-cosmos.pages.dev

View logs

[Copilot]

Pull Request Overview

This PR introduces support for copying a user-defined manifest.json when running tasks locally by leveraging ProjectConfig.manifest_path.

• Adds a new manifest_filepath parameter to local operators and propagates it through the converter
• Implements copy_manifest_file_if_exists in the DBT project module with corresponding unit tests
• Updates the local operator clone logic and adds an integration test to verify manifest copying

Reviewed Changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/operators/test_local.py	Added integration test to assert the manifest file is copied
tests/dbt/test_project.py	Added unit tests for copy_manifest_file_if_exists
cosmos/operators/local.py	Introduced manifest_filepath param and invoke manifest copy
cosmos/dbt/project.py	Added copy_manifest_file_if_exists function
cosmos/converter.py	Passed manifest_path from ProjectConfig into operator args

Files not reviewed (1)

• docs/configuration/operator-args.rst: Language not supported

Comments suppressed due to low confidence (2)

cosmos/operators/local.py:180

• [nitpick] The manifest_filepath parameter name differs from ProjectConfig.manifest_path; consider renaming it to manifest_path for consistency across your API.

manifest_filepath: str = "",

cosmos/operators/local.py:469

• copy_manifest_file_if_exists logs via the module-level logger, not the operator's logger; to ensure Airflow captures it (and the integration test sees it), log through self.log.info or accept a logger parameter.

copy_manifest_file_if_exists(self.manifest_filepath, Path(tmp_dir_path))

[pankajkoti]

The implementation looks good to me

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.71%. Comparing base (eb8114a) to head (3b6d504).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1772      +/-   ##
==========================================
+ Coverage   92.91%   97.71%   +4.80%     
==========================================
  Files          84       84              
  Lines        5262     5262              
==========================================
+ Hits         4889     5142     +253     
+ Misses        373      120     -253     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tuantran0910]

Hi @tatiana, can you provide me explanation why previously Cosmos was ignoring the user-defined manifest file while executing tasks ? I have read this PR but still not understand why the old way - defining manifest_file_path in ProjectConfig not works ?

[tatiana]

@tuantran0910 I believe it was a bug introduced through out the history of the project.

The manifest file was originally used in Cosmos for DAG rendering - not intentionally used for task execution

Historically, the first implementation of LoadMode.LOCAL would attempt to run dbt commands from the original dbt folder. However, this led to two issues:

• it didn't work when the filesystem was read-only
• it sometimes failed when running concurrent dbt commands that referenced the same target folder.

We attempted to configure different target & logs folders per Cosmos worker process, but there was an inconsistent way to accomplish this in different versions of dbt-core (flags, environment variable, in some cases changing the dbt_project.yml itself) - and we noticed dbt sometimes generated files outside of these two directories, including dbt_packages - which can only be customised via the dbt_project.yml'. We did not want to change the end-users dbt_project.yml`.

The next implementation consisted in copying the whole original dbt project directory into a temporary folder, from where Cosmos would execute the dbt commands - one temporary folder per task. End users reported issues with high disk utilization and slowness during the full copy.

To address this feedback, Cosmos started creating symbolic links from the temporary folder to the original one for some folders (e.g. for seeds& models folders), and would create an empty dedicated target folder per temporary folder.

This changed a bit over time, when we started supporting partial parsing files, and more recently when we introduced support to incremental dbt deps - but the manifest was still not being copied for task execution - until this PR.

This changed slightly

[tuantran0910]

Thank you @tatiana very much for the details explanation. I got it.