Fix the async execution mode read sql files for dbt packages by pankajastro · Pull Request #1588 · astronomer/astronomer-cosmos

pankajastro · 2025-03-04T18:03:14Z

closes: #1585

This PR modifies the DbtNode to include the packages,
allowing us to correctly construct the path when reading
the generated SQL files. In DBT projects with dbt_packages,
the dbt run command generates SQL files within the respective
dbt_packages folder inside the target/run directory, instead of
the main project folder.

With Setup task

Without Setup task

cloudflare-workers-and-pages · 2025-03-04T18:03:41Z

Deploying astronomer-cosmos with Cloudflare Pages

Latest commit:	`43da5c7`
Status:	✅ Deploy successful!
Preview URL:	https://c1206954.astronomer-cosmos.pages.dev
Branch Preview URL:	https://fix-read-path.astronomer-cosmos.pages.dev

View logs

netlify · 2025-03-04T18:03:42Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`43da5c7`
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/67d06e09cfa7e5000873a6e8

codecov · 2025-03-05T18:06:48Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.46%. Comparing base (b04717c) to head (43da5c7).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1588      +/-   ##
==========================================
+ Coverage   97.36%   97.46%   +0.10%     
==========================================
  Files          80       80              
  Lines        4934     4937       +3     
==========================================
+ Hits         4804     4812       +8     
+ Misses        130      125       -5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

tatiana

Hi @pankajastro , this is looking great, thank you very much!
Please, cover the areas that Codecov reported are lacking tests.

tatiana

Great work! Thank you very much for fixing this customer issue, @pankajastro!

closes: #1585 This PR modifies the DbtNode to include the packages, allowing us to correctly construct the path when reading the generated SQL files. In DBT projects with dbt_packages, the dbt run command generates SQL files within the respective dbt_packages folder inside the target/run directory, instead of the main project folder. <img width="1667" alt="Screenshot 2025-03-06 at 12 01 51 AM" src="https://github.com/user-attachments/assets/911e0859-327f-49bf-a081-4da7003d7817" /> **With Setup task** <img width="1687" alt="Screenshot 2025-03-06 at 12 02 58 AM" src="https://github.com/user-attachments/assets/c3c1f066-b19f-4bdd-9358-779845a32f8b" /> **Without Setup task** <img width="1688" alt="Screenshot 2025-03-06 at 12 03 34 AM" src="https://github.com/user-attachments/assets/b0bb21bd-e0d1-45a4-90ea-cb4857630318" /> (cherry picked from commit b309dac)

@pankajkoti

Bug Fixes * Fix import error in dbt bigquery adapter mock for ``dbt-bigquery<1.8`` for ``ExecutionMode.AIRFLOW_ASYNC`` by @pankajkoti in #1548 * Fix ``operator_args`` override configuration by @ghjklw in #1558 * Fix missing ``install_dbt_deps`` in ``ProjectConfig`` ``__init__`` method by @ghjklw in #1556 * Fix dbt project parsing ``dbt_vars`` behavior passed via ``operator_args`` by @AlexandrKhabarov in #1543 * Avoid reading the connection during DAG parsing of the async BigQuery operator by @joppevos in #1582 * Fix: Workaround to incorrectly raised ``gcsfs.retry.HttpError`` (Invalid Credentials, 401) by @tatiana in #1598 * Fix the async execution mode read sql files for dbt packages by @pankajastro in #1588 * Improve BQ async error handling by @tatiana in #1597 * Fix path selector when ``manifest.json`` is created using MS Windows by @tatiana in #1601 * Fix log that prints 'Total filtered nodes' by @tatiana in #1603 * Fix select behaviour using ``LoadMode.MANIFEST`` and a path with star by @tatiana in #1602 * Support ``on_warning_callback`` with ``TestBehavior.BUILD`` and ``ExecutionMode.LOCAL`` by @corsettigyg in #1571 * Fix ``DbtRunLocalOperator.partial()`` support by @tatiana @ashb in #1609 * fix: ``container_name`` is null for ecs integration by @nicor88 in #1592 Docs * Improve MWAA getting-started docs by removing unused imports by @jx2lee in #1562 Others * Disable ``example_cosmos_dbt_build.py`` DAG in CI by @pankajastro in #1567 * Upgrade GitHub Actions Ubuntu version by @tatiana in #1561 * Update GitHub bug issue template by @pankajastro in #1586 * Enable DAG ``example_cosmos_dbt_build.py`` in CI by @pankajastro in #1573 * Run async DAG in DAG without setup/teardown task by @pankajastro in #1599 * Add test case that fully covers recent select issue by @tatiana in #1604 * Add CI job to test multiple dbt versions for the async DAG by @pankajkoti in #1535 * Improve unit tests speed from 89s to 14s by @tatiana in #1600 * Pre-commit updates: #1560, #1583, #1596 Closes: #1550 Mergeable version of #1607 Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com> Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

pankajastro temporarily deployed to internal March 4, 2025 18:03 — with GitHub Actions Inactive

pankajastro added this to the Cosmos 1.9.1 milestone Mar 4, 2025

pankajastro temporarily deployed to internal March 5, 2025 09:04 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal March 5, 2025 11:26 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal March 5, 2025 17:24 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal March 5, 2025 17:32 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal March 5, 2025 17:40 — with GitHub Actions Inactive

pankajastro force-pushed the fix_read_path branch from c2b8b6d to ab188e5 Compare March 5, 2025 17:45

pankajastro temporarily deployed to internal March 5, 2025 17:46 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal March 5, 2025 18:14 — with GitHub Actions Inactive

pankajastro marked this pull request as ready for review March 5, 2025 18:34

dosubot Bot added size:S This PR changes 10-29 lines, ignoring generated files. area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc dbt:run Primarily related to dbt run command or functionality labels Mar 5, 2025

tatiana reviewed Mar 10, 2025

View reviewed changes

dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Mar 11, 2025

pankajastro temporarily deployed to internal March 11, 2025 11:02 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal March 11, 2025 17:01 — with GitHub Actions Inactive

pankajastro added 8 commits March 11, 2025 22:36

Fix the async execution mode read sql files for dbt packages

63ed8f5

Fix

8296fd6

Fix unit tests

87496be

more debug

fef6df5

more debug

bd27e83

more debug

a13168b

Update version

fad7707

Add tests

e7cddef

pankajastro force-pushed the fix_read_path branch from 1c60e1e to e7cddef Compare March 11, 2025 17:06

pankajastro temporarily deployed to internal March 11, 2025 17:06 — with GitHub Actions Inactive

Add tests

43da5c7

pankajastro temporarily deployed to internal March 11, 2025 17:08 — with GitHub Actions Inactive

pankajastro requested a review from tatiana March 11, 2025 17:40

tatiana approved these changes Mar 12, 2025

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label Mar 12, 2025

tatiana merged commit b309dac into main Mar 12, 2025

tatiana deleted the fix_read_path branch March 12, 2025 14:03

pankajkoti mentioned this pull request Mar 13, 2025

Release 1.9.1 #1607

Closed

tatiana mentioned this pull request Mar 13, 2025

Release 1.9.1 changelog #1612

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix the async execution mode read sql files for dbt packages#1588

Fix the async execution mode read sql files for dbt packages#1588
tatiana merged 9 commits into
mainfrom
fix_read_path

pankajastro commented Mar 4, 2025 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Mar 4, 2025 •

edited

Loading

Uh oh!

netlify Bot commented Mar 4, 2025 •

edited

Loading

Uh oh!

codecov Bot commented Mar 5, 2025 •

edited

Loading

Uh oh!

tatiana left a comment

Uh oh!

tatiana left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

pankajastro commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloudflare-workers-and-pages Bot commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying astronomer-cosmos with Cloudflare Pages

Uh oh!

netlify Bot commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Uh oh!

codecov Bot commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pankajastro commented Mar 4, 2025 •

edited

Loading

cloudflare-workers-and-pages Bot commented Mar 4, 2025 •

edited

Loading

netlify Bot commented Mar 4, 2025 •

edited

Loading

codecov Bot commented Mar 5, 2025 •

edited

Loading