Allow multiple callbacks by corsettigyg · Pull Request #1693 · astronomer/astronomer-cosmos

corsettigyg · 2025-04-24T19:14:20Z

Description

During some experimentation with callbacks I have noticed that we can only use a single callback per time, which can be very limiting if I want to add custom callbacks together with the default ones provided by cosmos.

This PR is very simple and aims to allow multiple callbacks to be passed as a list of functions, that will then be executed sequentially.

Related Issue(s)

Breaking Change?

Should not be

Checklist

I have made corresponding changes to the documentation (if required)
I have added tests that prove my fix is effective or that my feature works

netlify · 2025-04-24T19:14:38Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`940d0d8`
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/6810990f81d7c90008df2c3f

Copilot

Pull Request Overview

This pull request expands the functionality of callbacks by allowing multiple callbacks to be provided and executed sequentially.

Modified the callback parameter to accept either a single callback function or a list of functions.
Updated the post-execution logic to handle and iterate through a list of callbacks.

codecov · 2025-04-24T20:01:34Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.51%. Comparing base (441f21a) to head (940d0d8).
Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1693      +/-   ##
==========================================
+ Coverage   90.70%   97.51%   +6.81%     
==========================================
  Files          83       83              
  Lines        5174     5072     -102     
==========================================
+ Hits         4693     4946     +253     
+ Misses        481      126     -355

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

pankajkoti

hi @corsettigyg , thanks for adding this. It looks very helpful.

Would you please be able to add some test for the missing coverage report?

corsettigyg · 2025-04-25T14:29:00Z

@pankajkoti tried adding the tests

pankajkoti

LGTM. Thanks for adding the test @corsettigyg . I will wait for a while before merging the PR in case some else would like to take a review of the PR too. We're aiming to release Cosmos 1.10 sometime next week and would include this change in the release

tatiana

Looks great, @corsettigyg , thank you very much for adding this new feature in a way it is backwards compatible!

I added a minor feedback that is not a blocker for this PR to be merged - just an advice for future PRs:
https://github.com/astronomer/astronomer-cosmos/pull/1693/files#r2065890959

During some experimentation with [callbacks](https://astronomer.github.io/astronomer-cosmos/configuration/callbacks.html#example-using-callbacks-with-a-single-operator) I have noticed that we can only use a single callback per time, which can be very limiting if I want to add custom callbacks together with the default ones provided by cosmos. This PR is very simple and aims to allow multiple callbacks to be passed as a list of functions, that will then be executed sequentially.

@tatiana

Features * Airflow 3 support * Support running ``dbt deps`` incrementally to pre-defined ``dbt_packages`` by @tatiana in #1668 and #1670 * Add ``DuckDB`` profile mapping by @prithvijitguha and @pankajastro in #1553 * Implement DBT exposure selector by ghjklw #1717 Bug Fixes * Fix ``test_indirect_selection`` flag to be propagated in case of ``TestBehavior.BUILD`` by @corsettigyg in #1663 * Fix ``select`` clause in the case of detached tests by @anyapriya in #1680 * Operator argument fixes by @johnhoran in #1648 Airflow 3 Support * Support rendering DbtDag in Airflow 3 by @tatiana and @ashb in #1657 * Refactor Rendered Task Instance Fields (RTIF) handling for Airflow 2.x and 3.x by @pankajkoti in #1661 * Run cosmos operator in Airflow 3 by @pankajastro in #1642 * Fix ``python_virtualenv.prepare_env`` top-level import for Airflow 3 by @pankajkoti in #1678 * Fix Variable not found issue in Airflow 3 by @tatiana in #1684 * Disable CosmosPlugin on Airflow 3 setup by @pankajkoti in #1692, #1698 * Use ``schedule`` param in example DAGs instead of the 2.10 deprecated and 3.0 removed ``schedule_interval`` by @pankajkoti in #1701 * Ensure ``virtualenv_dir`` path exists by @pankajkoti in #1724 * Support emitting Assets with Airflow 3 by @tatiana in #1713 * Add docs on Airflow 3 compatibility by @pankajkoti and @tatiana in #1731 * Introduce, test and document asset/dataset breaking change by @tatiana in #1672 * Improve dataset/asset driven scheduling documentation by @tatiana in #1729 Enhancements * Allow multiple callbacks by @corsettigyg #1693 * Refactor kubernetes warning callback handling by @canbekley in #1681 Documentation * Add documentation related to ``copy_dbt_packages`` by @tatiana in #1671 * Make wording and command consistent in the contributing doc by @pankajkoti in #1697 * Add MonteCarlo callback example for importing dbt artifacts by @corsettigyg #1695 * Change async feature to be non-experimental by @tatiana in #1732 Others * Add sample ``dbt_packages`` to validate incremental ``dbt deps`` by @tatiana in #1669 * Add kubernetes execution mode example in Airflow 3 by @pankajastro in #1667 * Check only major version until Airflow 3 stable release by @pankajastro in #1665 * Install Airflow from main branch by @pankajastro in #1660 * Add dev tool for Airflow 3 by @pankajastro and @tatiana in #1627 * Improve Airflow 3 tooling by @pankajastro in #1656 * Skip associating ``openlineage_events_completes`` to ``ti`` in Airflow 3 by @pankajkoti in #1662 * Add .gitignore file for the scripts/airflow3 directory by @pankajkoti in #1658 * Remove ``original_jaffle_shop`` dbt project by @pankajkoti in #1676 * Fix or ignore type check error by @pankajastro in #1687 * Run virtualenv example with Airflow 3 tooling by @pankajastro in #1686 * Enable running setup/teardown tasks with Async execution DAG with Airflow 3 tooling by @pankajastro in #1696 * Enable integration tests for the DuckDB adapter by @pankajastro in #1699 * Add Airflow 3 tests matrix entries in CI by @pankajkoti in #1646 * Use a different way to get tasks count for asserting test_perf_dag by @pankajkoti in #1714 * Reinstall Airflow 3 dependency on ``pydantic>=2.11`` for dbt adapter versions 1.6 & 1.9 by @pankajkoti in #1715 * Fix outdated ``echo`` in Airflow 3 tooling script #1700 * Add files not needed for git tracking to .gitignore by @pankajkoti in #1723 * Use latest minor versions for dbt adapters to get in compatibility fixes by @pankajkoti in #1719 * Fix Airflow 3 tests raising generate_run_id() takes 0 positional arguments by @tatiana in #1725 * Fix dataset tests failing in Airflow 3 by @tatiana in #1716 * Enable example DAGs to run in CI that were disabled in PR #1646 by @pankajkoti in #1726 * Pre-commit updates: #1666, #1653, #1641, #1682, #1720 Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com> --------- Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

@corsettigyg

[Giovanni Corsetti Silva](https://www.linkedin.com/in/giovanni-corsetti/) (@corsettigyg @CorsettiS @giovannicorsetti) is a Core Data Platform Data Engineer at [Get Your Guide](https://www.getyourguide.com/). He's a Brazilian based in Berlin (Germany) and has been an early adopter of Cosmos, using it regularly throughout his career. Not only has he been using Cosmos since the early stages, but he has consistently improved Cosmos since April 2023: <img width="494" height="249" alt="Screenshot 2025-09-17 at 16 39 49" src="https://github.com/user-attachments/assets/ff9b0055-7df6-47a4-8efa-83a30d5aee99" /> ] m. , <img width="492" height="252" alt="Screenshot 2025-09-17 at 16 40 11" src="https://github.com/user-attachments/assets/8402d883-eff4-46aa-9ae7-afe35bd39a6e" /> <img width="499" height="256" alt="Screenshot 2025-09-17 at 16 39 23" src="https://github.com/user-attachments/assets/c6b84245-aae1-46e6-99e3-552553619dd9" /> His contributions include new features, enhancements, bug fixes and also improvements on the local dev setup, as it can see in some of the many contributions he's done to the project so far: * #1787 * #1761 * #1695 * #1693 * #1663 * #1571 * #1449 * #1099 * #1091 * #419 * #402 * #355 * #354 * #225 Additionally, he has been interacting with users in the #airflow-dbt Slack channel in a very collaborative and supportive way. We want to promote him as a Cosmos committer and maintainer for all these, recognising his constant efforts and achievements towards our community. Thank you very much, @corsettigyg! From this moment, you have write permissions on the repo: <img width="884" height="93" alt="Screenshot 2025-09-18 at 14 34 07" src="https://github.com/user-attachments/assets/abcc8c42-87cf-4b32-a7be-81127190d164" />

Copilot AI review requested due to automatic review settings April 24, 2025 19:14

dosubot Bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Apr 24, 2025

corsettigyg had a problem deploying to external April 24, 2025 19:14 — with GitHub Actions Error

Copilot AI reviewed Apr 24, 2025

View reviewed changes

Comment thread cosmos/operators/local.py Outdated

corsettigyg temporarily deployed to external April 24, 2025 19:36 — with GitHub Actions Inactive

pankajkoti reviewed Apr 25, 2025

View reviewed changes

pankajkoti self-assigned this Apr 25, 2025

pankajkoti added this to the Cosmos 1.10.0 milestone Apr 25, 2025

dosubot Bot added size:S This PR changes 10-29 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels Apr 25, 2025

corsettigyg had a problem deploying to external April 25, 2025 14:06 — with GitHub Actions Error

dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Apr 25, 2025

pre-commit-ci Bot had a problem deploying to external April 25, 2025 14:06 Error

corsettigyg temporarily deployed to external April 25, 2025 14:08 — with GitHub Actions Inactive

pankajkoti approved these changes Apr 25, 2025

View reviewed changes

corsettigyg and others added 4 commits April 29, 2025 11:01

allow multiple callback functions

387543c

remove comment (unnecessary)

0997c7d

add tests

f119016

🎨 [pre-commit.ci] Auto format from pre-commit.com hooks

705f29d

pankajkoti force-pushed the main branch from c9cfd87 to 705f29d Compare April 29, 2025 05:31

pankajkoti temporarily deployed to external April 29, 2025 05:31 — with GitHub Actions Inactive

tatiana reviewed Apr 29, 2025

View reviewed changes

Comment thread tests/operators/test_local.py

tatiana approved these changes Apr 29, 2025

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label Apr 29, 2025

Merge branch 'main' into main

940d0d8

tatiana temporarily deployed to external April 29, 2025 09:17 — with GitHub Actions Inactive

pankajkoti merged commit a4e0805 into astronomer:main Apr 29, 2025
92 checks passed

pankajkoti mentioned this pull request Apr 30, 2025

Release 1.10.0 #1672

Merged

tatiana mentioned this pull request Sep 18, 2025

Promote @corsettigyg to committer #1985

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow multiple callbacks#1693

Allow multiple callbacks#1693
pankajkoti merged 5 commits into
astronomer:mainfrom
corsettigyg:main

corsettigyg commented Apr 24, 2025 •

edited

Loading

Uh oh!

netlify Bot commented Apr 24, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

codecov Bot commented Apr 24, 2025 •

edited

Loading

Uh oh!

pankajkoti left a comment

Uh oh!

corsettigyg commented Apr 25, 2025

Uh oh!

pankajkoti left a comment

Uh oh!

Uh oh!

tatiana left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

corsettigyg commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Breaking Change?

Checklist

Uh oh!

netlify Bot commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

codecov Bot commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pankajkoti left a comment

Choose a reason for hiding this comment

Uh oh!

corsettigyg commented Apr 25, 2025

Uh oh!

pankajkoti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

corsettigyg commented Apr 24, 2025 •

edited

Loading

netlify Bot commented Apr 24, 2025 •

edited

Loading

codecov Bot commented Apr 24, 2025 •

edited

Loading