Support on_warning_callback with TestBehavior.BUILD and ExecutionMode.LOCAL#1571
Conversation
remove args["on_warning_callback"] = on_warning_callback unused
✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.
|
|
it seems that the CI is failing due to a different error (#1567) |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1571 +/- ##
=======================================
Coverage 97.34% 97.35%
=======================================
Files 80 80
Lines 4903 4920 +17
=======================================
+ Hits 4773 4790 +17
Misses 130 130 ☔ View full report in Codecov by Sentry. |
add DbtBuildLocalOperator test on_warning_callback
There was a problem hiding this comment.
@corsettigyg thanks a lot for improving this.
Ideally, we'd also have feature parity / fix this in the other executors, as an example:
- https://github.com/astronomer/astronomer-cosmos/blob/main/cosmos/operators/kubernetes.py#L120-L128 (which even has a generic class to handle warnings)
- https://github.com/astronomer/astronomer-cosmos/blob/main/cosmos/operators/docker.py#L96-L104
- https://github.com/astronomer/astronomer-cosmos/blob/main/cosmos/operators/gcp_cloud_run_job.py#L126-L134
- https://github.com/astronomer/astronomer-cosmos/blob/main/cosmos/operators/azure_container_instance.py#L104-L112
astronomer-cosmos/cosmos/operators/aws_ecs.py
Line 122 in 0811e46
So their build operators also can receive on_warning_callback.
I understand these changes are outside the scope of the current PR, but could you please log a GitHub issue so we can keep track of them?
I'm happy for us to merge this PR once the checks pass.
on_warning_callback with TestBehavior.BUILD and ExecutionMode.LOCAL
…nMode.LOCAL` (#1571) As of now, when we set TestBehavior.BUILD, we are not leveraging the method on_warning_callback that is available for Test nodes and Source Nodes. I have added the parsing to DbtBuildLocalOperator in order to fix it. I tested it locally and I got positive results Related: #1569 (cherry picked from commit ddea39c)
Bug Fixes * Fix import error in dbt bigquery adapter mock for ``dbt-bigquery<1.8`` for ``ExecutionMode.AIRFLOW_ASYNC`` by @pankajkoti in #1548 * Fix ``operator_args`` override configuration by @ghjklw in #1558 * Fix missing ``install_dbt_deps`` in ``ProjectConfig`` ``__init__`` method by @ghjklw in #1556 * Fix dbt project parsing ``dbt_vars`` behavior passed via ``operator_args`` by @AlexandrKhabarov in #1543 * Avoid reading the connection during DAG parsing of the async BigQuery operator by @joppevos in #1582 * Fix: Workaround to incorrectly raised ``gcsfs.retry.HttpError`` (Invalid Credentials, 401) by @tatiana in #1598 * Fix the async execution mode read sql files for dbt packages by @pankajastro in #1588 * Improve BQ async error handling by @tatiana in #1597 * Fix path selector when ``manifest.json`` is created using MS Windows by @tatiana in #1601 * Fix log that prints 'Total filtered nodes' by @tatiana in #1603 * Fix select behaviour using ``LoadMode.MANIFEST`` and a path with star by @tatiana in #1602 * Support ``on_warning_callback`` with ``TestBehavior.BUILD`` and ``ExecutionMode.LOCAL`` by @corsettigyg in #1571 * Fix ``DbtRunLocalOperator.partial()`` support by @tatiana @ashb in #1609 * fix: ``container_name`` is null for ecs integration by @nicor88 in #1592 Docs * Improve MWAA getting-started docs by removing unused imports by @jx2lee in #1562 Others * Disable ``example_cosmos_dbt_build.py`` DAG in CI by @pankajastro in #1567 * Upgrade GitHub Actions Ubuntu version by @tatiana in #1561 * Update GitHub bug issue template by @pankajastro in #1586 * Enable DAG ``example_cosmos_dbt_build.py`` in CI by @pankajastro in #1573 * Run async DAG in DAG without setup/teardown task by @pankajastro in #1599 * Add test case that fully covers recent select issue by @tatiana in #1604 * Add CI job to test multiple dbt versions for the async DAG by @pankajkoti in #1535 * Improve unit tests speed from 89s to 14s by @tatiana in #1600 * Pre-commit updates: #1560, #1583, #1596 Closes: #1550 Mergeable version of #1607 Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com> Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
[Giovanni Corsetti Silva](https://www.linkedin.com/in/giovanni-corsetti/) (@corsettigyg @CorsettiS @giovannicorsetti) is a Core Data Platform Data Engineer at [Get Your Guide](https://www.getyourguide.com/). He's a Brazilian based in Berlin (Germany) and has been an early adopter of Cosmos, using it regularly throughout his career. Not only has he been using Cosmos since the early stages, but he has consistently improved Cosmos since April 2023: <img width="494" height="249" alt="Screenshot 2025-09-17 at 16 39 49" src="https://github.com/user-attachments/assets/ff9b0055-7df6-47a4-8efa-83a30d5aee99" /> ] m. , <img width="492" height="252" alt="Screenshot 2025-09-17 at 16 40 11" src="https://github.com/user-attachments/assets/8402d883-eff4-46aa-9ae7-afe35bd39a6e" /> <img width="499" height="256" alt="Screenshot 2025-09-17 at 16 39 23" src="https://github.com/user-attachments/assets/c6b84245-aae1-46e6-99e3-552553619dd9" /> His contributions include new features, enhancements, bug fixes and also improvements on the local dev setup, as it can see in some of the many contributions he's done to the project so far: * #1787 * #1761 * #1695 * #1693 * #1663 * #1571 * #1449 * #1099 * #1091 * #419 * #402 * #355 * #354 * #225 Additionally, he has been interacting with users in the #airflow-dbt Slack channel in a very collaborative and supportive way. We want to promote him as a Cosmos committer and maintainer for all these, recognising his constant efforts and achievements towards our community. Thank you very much, @corsettigyg! From this moment, you have write permissions on the repo: <img width="884" height="93" alt="Screenshot 2025-09-18 at 14 34 07" src="https://github.com/user-attachments/assets/abcc8c42-87cf-4b32-a7be-81127190d164" />
Description
as of now, when we set TestBehavior.BUILD, we are not leveraging the method on_warning_callback that is available for Test nodes and Source Nodes. I have added the parsing to DbtBuildLocalOperator in order to fix it. I tested it locally and I got positive results
Related Issue(s)
#1569
Breaking Change?
Should not be
Checklist