Skip to content

Support on_warning_callback with TestBehavior.BUILD and ExecutionMode.LOCAL#1571

Merged
tatiana merged 11 commits into
astronomer:mainfrom
corsettigyg:main
Feb 28, 2025
Merged

Support on_warning_callback with TestBehavior.BUILD and ExecutionMode.LOCAL#1571
tatiana merged 11 commits into
astronomer:mainfrom
corsettigyg:main

Conversation

@corsettigyg
Copy link
Copy Markdown
Collaborator

@corsettigyg corsettigyg commented Feb 27, 2025

Description

as of now, when we set TestBehavior.BUILD, we are not leveraging the method on_warning_callback that is available for Test nodes and Source Nodes. I have added the parsing to DbtBuildLocalOperator in order to fix it. I tested it locally and I got positive results

Related Issue(s)

#1569

Breaking Change?

Should not be

Checklist

  • I have made corresponding changes to the documentation (if required)
  • I have added tests that prove my fix is effective or that my feature works

@dosubot dosubot Bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Feb 27, 2025
@netlify
Copy link
Copy Markdown

netlify Bot commented Feb 27, 2025

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit 5867fd4
🔍 Latest deploy log https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/67c11983253c2f0008878dde

@dosubot dosubot Bot added area:parsing Related to parsing DAG/DBT improvement, issues, or fixes dbt:build Primarily related to dbt build command or functionality execution:local Related to Local execution environment labels Feb 27, 2025
@corsettigyg
Copy link
Copy Markdown
Collaborator Author

corsettigyg commented Feb 27, 2025

it seems that the CI is failing due to a different error (#1567)

@codecov
Copy link
Copy Markdown

codecov Bot commented Feb 27, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.35%. Comparing base (0811e46) to head (5867fd4).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1571   +/-   ##
=======================================
  Coverage   97.34%   97.35%           
=======================================
  Files          80       80           
  Lines        4903     4920   +17     
=======================================
+ Hits         4773     4790   +17     
  Misses        130      130           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Copy Markdown
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@corsettigyg thanks a lot for improving this.

Ideally, we'd also have feature parity / fix this in the other executors, as an example:

So their build operators also can receive on_warning_callback.

I understand these changes are outside the scope of the current PR, but could you please log a GitHub issue so we can keep track of them?

I'm happy for us to merge this PR once the checks pass.

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Feb 28, 2025
@corsettigyg corsettigyg changed the title make on_warning_callback functional with TestBehavior.BUILD make on_warning_callback functional with TestBehavior.BUILD for Local EM Feb 28, 2025
@corsettigyg
Copy link
Copy Markdown
Collaborator Author

@tatiana Good catch ! I renamed the name of this PR to specify that it only accommodates the LocalOperator EM. There is an issue logged here that I have created.

Since the CI is all green, happy to contribute :)

@tatiana tatiana changed the title make on_warning_callback functional with TestBehavior.BUILD for Local EM Support on_warning_callback with TestBehavior.BUILD and ExecutionMode.LOCAL Feb 28, 2025
@tatiana tatiana merged commit ddea39c into astronomer:main Feb 28, 2025
pankajkoti pushed a commit that referenced this pull request Mar 13, 2025
…nMode.LOCAL` (#1571)

As of now, when we set TestBehavior.BUILD, we are not leveraging the
method on_warning_callback that is available for Test nodes and Source
Nodes. I have added the parsing to DbtBuildLocalOperator in order to fix
it. I tested it locally and I got positive results

Related: #1569
(cherry picked from commit ddea39c)
pankajkoti added a commit that referenced this pull request Mar 13, 2025
@pankajkoti pankajkoti mentioned this pull request Mar 13, 2025
@tatiana tatiana mentioned this pull request Mar 13, 2025
tatiana added a commit that referenced this pull request Mar 17, 2025
Bug Fixes

* Fix import error in dbt bigquery adapter mock for ``dbt-bigquery<1.8``
for ``ExecutionMode.AIRFLOW_ASYNC`` by @pankajkoti in #1548
* Fix ``operator_args`` override configuration by @ghjklw in #1558
* Fix missing ``install_dbt_deps`` in ``ProjectConfig`` ``__init__``
method by @ghjklw in #1556
* Fix dbt project parsing ``dbt_vars`` behavior passed via
``operator_args`` by @AlexandrKhabarov in #1543
* Avoid reading the connection during DAG parsing of the async BigQuery
operator by @joppevos in #1582
* Fix: Workaround to incorrectly raised ``gcsfs.retry.HttpError``
(Invalid Credentials, 401) by @tatiana in #1598
* Fix the async execution mode read sql files for dbt packages by
@pankajastro in #1588
* Improve BQ async error handling by @tatiana in #1597
* Fix path selector when ``manifest.json`` is created using MS Windows
by @tatiana in #1601
* Fix log that prints 'Total filtered nodes' by @tatiana in #1603
* Fix select behaviour using ``LoadMode.MANIFEST`` and a path with star
by @tatiana in #1602
* Support ``on_warning_callback`` with ``TestBehavior.BUILD`` and
``ExecutionMode.LOCAL`` by @corsettigyg in #1571
* Fix ``DbtRunLocalOperator.partial()`` support by @tatiana @ashb in
#1609
* fix: ``container_name`` is null for ecs integration by @nicor88 in
#1592

Docs

* Improve MWAA getting-started docs by removing unused imports by
@jx2lee in #1562

Others

* Disable ``example_cosmos_dbt_build.py`` DAG in CI by @pankajastro in
#1567
* Upgrade GitHub Actions Ubuntu version by @tatiana in #1561
* Update GitHub bug issue template by @pankajastro in #1586
* Enable DAG ``example_cosmos_dbt_build.py`` in CI by @pankajastro in
#1573
* Run async DAG in DAG without setup/teardown task by @pankajastro in
#1599
* Add test case that fully covers recent select issue by @tatiana in
#1604
* Add CI job to test multiple dbt versions for the async DAG by
@pankajkoti in #1535
* Improve unit tests speed from 89s to 14s by @tatiana in #1600
* Pre-commit updates: #1560, #1583, #1596


Closes: #1550

Mergeable version of
#1607

Co-authored-by: Pankaj Singh
<98807258+pankajastro@users.noreply.github.com>
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
tatiana added a commit that referenced this pull request Sep 18, 2025
[Giovanni Corsetti
Silva](https://www.linkedin.com/in/giovanni-corsetti/) (@corsettigyg
@CorsettiS @giovannicorsetti) is a Core Data Platform Data Engineer at
[Get Your Guide](https://www.getyourguide.com/). He's a Brazilian based
in Berlin (Germany) and has been an early adopter of Cosmos, using it
regularly throughout his career.

Not only has he been using Cosmos since the early stages, but he has
consistently improved Cosmos since April 2023:

<img width="494" height="249" alt="Screenshot 2025-09-17 at 16 39 49"
src="https://github.com/user-attachments/assets/ff9b0055-7df6-47a4-8efa-83a30d5aee99"
/>
] m. ,
<img width="492" height="252" alt="Screenshot 2025-09-17 at 16 40 11"
src="https://github.com/user-attachments/assets/8402d883-eff4-46aa-9ae7-afe35bd39a6e"
/>

<img width="499" height="256" alt="Screenshot 2025-09-17 at 16 39 23"
src="https://github.com/user-attachments/assets/c6b84245-aae1-46e6-99e3-552553619dd9"
/>

His contributions include new features, enhancements, bug fixes and also
improvements on the local dev setup, as it can see in some of the many
contributions he's done to the project so far:

* #1787
* #1761
* #1695 
* #1693 
* #1663
* #1571
* #1449
* #1099
* #1091
* #419
* #402
* #355 
* #354 
* #225 

Additionally, he has been interacting with users in the #airflow-dbt
Slack channel in a very collaborative and supportive way.

We want to promote him as a Cosmos committer and maintainer for all
these, recognising his constant efforts and achievements towards our
community. Thank you very much, @corsettigyg!

From this moment, you have write permissions on the repo:
<img width="884" height="93" alt="Screenshot 2025-09-18 at 14 34 07"
src="https://github.com/user-attachments/assets/abcc8c42-87cf-4b32-a7be-81127190d164"
/>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:parsing Related to parsing DAG/DBT improvement, issues, or fixes dbt:build Primarily related to dbt build command or functionality execution:local Related to Local execution environment lgtm This PR has been approved by a maintainer size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants