Disable dbt static parser during Airflow task execution using dbt runner by pankajkoti · Pull Request #1760 · astronomer/astronomer-cosmos

pankajkoti · 2025-05-09T10:08:32Z

This PR adds support for conditionally applying the --no-static-parser dbt flag in Cosmos operators, ensuring it is included only when InvocationMode.DBT_RUNNER is used during task execution.

Rationale

Static Parser Issue: User reports and investigation revealed that, starting with Cosmos 1.9.0 (see PR #1484), using dbtRunner for both DAG parsing and task execution in Airflow 2.x can cause task hangs. This is due to dbt's static parser interacting poorly with Cosmos's use of temporary project directories, especially when the temp paths differ between parsing and execution.
Workaround: Adding the --no-static-parser flag when invoking dbtRunner during task execution avoids these hangs and ensures reliable operation. This flag is not needed (and should not be added) when using the subprocess invocation mode.

closes: #1751
related: #1750

Co-authored-by: Tatiana Al-Chueyr tatiana.alchueyr@gmail.com

netlify · 2025-05-09T10:08:38Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`b00ae62`
🔍 Latest deploy log	https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/682c4f9421ff420008db5e7e

cloudflare-workers-and-pages · 2025-05-09T10:09:07Z

Deploying astronomer-cosmos with Cloudflare Pages

Latest commit:	`b00ae62`
Status:	✅ Deploy successful!
Preview URL:	https://a125282d.astronomer-cosmos.pages.dev
Branch Preview URL:	https://disable-static-parser-dbtrun.astronomer-cosmos.pages.dev

View logs

codecov · 2025-05-09T10:52:37Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.71%. Comparing base (81e248a) to head (b00ae62).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1760   +/-   ##
=======================================
  Coverage   97.71%   97.71%           
=======================================
  Files          84       84           
  Lines        5250     5252    +2     
=======================================
+ Hits         5130     5132    +2     
  Misses        120      120

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

updates: - [github.com/astral-sh/ruff-pre-commit: v0.11.8 → v0.11.9](astral-sh/ruff-pre-commit@v0.11.8...v0.11.9)  Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…ing local directory (#1740) Ensure remote target directory are created when copying files when using local directory. When configuring a remote target directory that points to a local path while using AIRFLOW ASYNC, like so: ```bash AIRFLOW__COSMOS__REMOTE_TARGET_PATH=/usr/local/airflow/cosmos AIRFLOW__COSMOS__REMOTE_TARGET_PATH_CONN_ID=file_default ``` We might face this issue: ```bash FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/airflow/cosmos/simple_dag_async__dbt_async/run/jaffle_shop/models/example/my_second_dbt_model.sql' ``` Closes #1739 Co-authored-by: Giovanni Corsetti <155465603+corsettigyg@users.noreply.github.com>

The feature introduced in #1670 (Support running `dbt deps` incrementally to pre-defined `dbt_packages` during task execution) did not work as expected if users had defined a custom path for `packages-install-path`. It only worked if the default (`dbt_packages` was being used. This PR aims to solve the issue.

updates: - [github.com/astral-sh/ruff-pre-commit: v0.11.9 → v0.11.10](astral-sh/ruff-pre-commit@v0.11.9...v0.11.10)  Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…DEBUG` (#1764) Recently, there have been some concerns that Cosmos may modify the `packages.yml` content, leading to errors. If users set `AIRFLOW__LOGGING__LOGGING_LEVEL=DEBUG`, they should now be able to confirm the content of the file. Example of log output: ``` [2025-05-12T10:52:38.183+0100] {local.py:481} DEBUG - Checking for the packages.yml dependencies file. [2025-05-12T10:52:38.184+0100] {local.py:484} DEBUG - Contents of the </var/folders/td/522y78v91d1f5wgh67mj3p0m0000gn/T/tmp_4q53rv2/packages.yml> dependencies file: packages: - package: dbt-labs/dbt_utils version: "1.1.1" ```

tatiana

@pankajkoti its great you found this alternative than setting SUBPROCESS as the standard InvocationMethod during task execution. I'm glad we have a path to avoid the hanging behaviour some customers were facing.

Would it be worth for us to add a follow-up ticket for us to review in the future if we can find alternatives to this? According to dbt there can be significant performance improvements by enabling this setting, and we may be able to find alternative solutions in the future for this problem

pankajkoti · 2025-05-20T11:17:29Z

Would it be worth for us to add a follow-up ticket for us to review in the future if we can find alternatives to this? According to dbt there can be significant performance improvements by enabling this setting, and we may be able to find alternative solutions in the future for this problem

Yes @tatiana, although I have not found any significant improvement with using static parsing, nor a degradation upon disabling it :), I have logged a follow-up ticket #1771 to investigate that

…ner (#1760) This PR adds support for conditionally applying the `--no-static-parser` dbt flag in Cosmos operators, ensuring it is included only when InvocationMode.DBT_RUNNER is used during task execution. **Static Parser Issue**: User reports and investigation revealed that, starting with Cosmos 1.9.0 (see PR #1484), using dbtRunner for both DAG parsing and task execution in Airflow 2.x can cause task hangs. This is due to dbt's static parser interacting poorly with Cosmos's use of temporary project directories, especially when the temp paths differ between parsing and execution. **Workaround**: Adding the `--no-static-parser` flag when invoking dbtRunner during task execution avoids these hangs and ensures reliable operation. This flag is not needed (and should not be added) when using the subprocess invocation mode. closes: #1751 related: #1750 Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com> (cherry picked from commit 15a8d91)

@tuantran0910

Bug Fixes * Fix ``full_refresh`` parameter in ``AIRFLOW_ASYNC`` ``ExecutionConfig`` mode by @tuantran0910 in #1738 * Fix dbt ls invocation method log message by @tatiana and @dstandish in #1749 * Ensure remote target directory is created when copying files when using local directory by @tuantran0910 and @corsettigyg in #1740 * Support custom ``packages-install-path`` by @tatiana in #1768 * Disable dbt static parser during Airflow task execution using dbt runner by @pankajkoti and @tatiana in #1760 * Fix ``ExecutionMode.LOCAL`` to leverage ``ProjectConfig.manifest_path`` by @tatiana in #1772 * Refactor ``AIRFLOW_ASYNC`` so that the path in the remote object store is specific per DAG run by @tuantran0910 in #1741 * Optimise memory usage with optional explicit imports by @pankajkoti and @tatiana in #1769 Documentation * Fix documentation rendering for ``use_dataset_airflow3_uri_standard`` by @pankajastro in #1742 * Correct custom callback example by @walter9388 in #1747 Others * Re-enable integration tests durations to troubleshoot performance degradation by @tatiana in #1735 * Run listener tests for Airflow 3 by @pankajastro in #1743 * Add Airflow 3 db files to ignore from git tracking by @pankajkoti in #1755 * Log contents of ``packages.yml`` when ``AIRFLOW__LOGGING__LOGGING_LEVEL=DEBUG`` by @tatiana in #1764 * Fix Airflow dependencies in the CI by @tatiana in #1773 * Pre-commit updates: #1744, #1765, #1770

@tuantran0910

Bug Fixes * Fix ``full_refresh`` parameter in ``AIRFLOW_ASYNC`` ``ExecutionConfig`` mode by @tuantran0910 in #1738 * Fix dbt ls invocation method log message by @tatiana and @dstandish in #1749 * Ensure remote target directory is created when copying files when using local directory by @tuantran0910 and @corsettigyg in #1740 * Support custom ``packages-install-path`` by @tatiana in #1768 * Disable dbt static parser during Airflow task execution using dbt runner by @pankajkoti and @tatiana in #1760 * Fix ``ExecutionMode.LOCAL`` to leverage ``ProjectConfig.manifest_path`` by @tatiana in #1772 * Refactor ``AIRFLOW_ASYNC`` so that the path in the remote object store is specific per DAG run by @tuantran0910 in #1741 * Optimise memory usage with optional explicit imports by @pankajkoti and @tatiana in #1769 Documentation * Fix documentation rendering for ``use_dataset_airflow3_uri_standard`` by @pankajastro in #1742 * Correct custom callback example by @walter9388 in #1747 Others * Re-enable integration tests durations to troubleshoot performance degradation by @tatiana in #1735 * Run listener tests for Airflow 3 by @pankajastro in #1743 * Add Airflow 3 db files to ignore from git tracking by @pankajkoti in #1755 * Log contents of ``packages.yml`` when ``AIRFLOW__LOGGING__LOGGING_LEVEL=DEBUG`` by @tatiana in #1764 * Fix Airflow dependencies in the CI by @tatiana in #1773 * Pre-commit updates: #1744, #1765, #1770 --------- (cherry picked from commit 430be00)

Reference #1751 (original report of dbt_runner task hangs) and #1760 (the fix that introduced the --no-static-parser injection) so readers can trace when and how the workaround was reached.

Disable dbt static parser during Airflow task execution using dbt runner

1837c32

pankajkoti temporarily deployed to internal May 9, 2025 10:08 — with GitHub Actions Inactive

pankajkoti mentioned this pull request May 19, 2025

Default RenderConfig’s InvocationMode to SUBPROCESS to avoid task hangs observed with dbt Runner on Airflow 2 #1750

Closed

pankajkoti commented May 20, 2025

View reviewed changes

Comment thread cosmos/__init__.py Outdated

pre-commit-ci Bot and others added 6 commits May 20, 2025 15:15

Add comment and tests

5df10c7

pankajkoti temporarily deployed to internal May 20, 2025 09:45 — with GitHub Actions Inactive

Update cosmos/__init__.py

46488de

pankajkoti temporarily deployed to internal May 20, 2025 09:46 — with GitHub Actions Inactive

pankajkoti and others added 2 commits May 20, 2025 15:16

Merge branch 'main' into disable-static-parser-dbtrunner-task-execution

245d231

🎨 [pre-commit.ci] Auto format from pre-commit.com hooks

b00ae62

pre-commit-ci Bot temporarily deployed to internal May 20, 2025 09:47 Inactive

pankajkoti commented May 20, 2025

View reviewed changes

Comment thread cosmos/operators/local.py

pankajkoti marked this pull request as ready for review May 20, 2025 10:33

dosubot Bot added the size:M This PR changes 30-99 lines, ignoring generated files. label May 20, 2025

pankajkoti requested review from pankajastro and tatiana May 20, 2025 10:33

tatiana approved these changes May 20, 2025

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label May 20, 2025

tatiana added this to the Cosmos 1.10.1 milestone May 20, 2025

pankajkoti mentioned this pull request May 20, 2025

Investigate safe reintroduction of dbt static parser during task execution with dbtRunner in Cosmos #1771

Open

tatiana merged commit 15a8d91 into main May 20, 2025
177 of 178 checks passed

tatiana deleted the disable-static-parser-dbtrunner-task-execution branch May 20, 2025 13:56

This was referenced May 21, 2025

Release 1.10.1 #1774

Merged

Release 1.10.1 (add changelog and version bump) #1775

Merged

pankajkoti mentioned this pull request Aug 19, 2025

Docs: Add disclaimer for static-parser when using execution mode local #1925

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable dbt static parser during Airflow task execution using dbt runner#1760

Disable dbt static parser during Airflow task execution using dbt runner#1760
tatiana merged 10 commits into
mainfrom
disable-static-parser-dbtrunner-task-execution

pankajkoti commented May 9, 2025 •

edited

Loading

Uh oh!

netlify Bot commented May 9, 2025 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented May 9, 2025 •

edited

Loading

Uh oh!

codecov Bot commented May 9, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment •

edited by pankajkoti

Loading

Uh oh!

pankajkoti commented May 20, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pankajkoti commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale

Uh oh!

netlify Bot commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Uh oh!

cloudflare-workers-and-pages Bot commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying astronomer-cosmos with Cloudflare Pages

Uh oh!

codecov Bot commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment • edited by pankajkoti Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pankajkoti commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pankajkoti commented May 9, 2025 •

edited

Loading

netlify Bot commented May 9, 2025 •

edited

Loading

cloudflare-workers-and-pages Bot commented May 9, 2025 •

edited

Loading

codecov Bot commented May 9, 2025 •

edited

Loading

tatiana left a comment •

edited by pankajkoti

Loading

pankajkoti commented May 20, 2025 •

edited

Loading