Skip to content

Allow to disable owner inheritance from dbt into airflow DAG owners#1787

Merged
tatiana merged 17 commits into
astronomer:mainfrom
CorsettiS:main
Jun 3, 2025
Merged

Allow to disable owner inheritance from dbt into airflow DAG owners#1787
tatiana merged 17 commits into
astronomer:mainfrom
CorsettiS:main

Conversation

@CorsettiS
Copy link
Copy Markdown
Contributor

Description

From slack
Hey everyone! :wave: I'm working on an Airflow DAG that uses DbtTaskGroup (Cosmos v1.10.1). I've noticed that the Airflow tasks are showing both the DAG owner (set in default_args) and the owner defined in my dbt model's _doc_.yml. Is there a way to configure Cosmos to not pull the owner from the dbt model? I'm aiming to keep the DAG owner clean. Any insights would be greatly

This PR introduces the flag disable_owner_inheritance which should make it possible

Related Issue(s)

Breaking Change?

No

Checklist

  • I have made corresponding changes to the documentation (if required)
  • I have added tests that prove my fix is effective or that my feature works

@netlify
Copy link
Copy Markdown

netlify Bot commented May 27, 2025

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit 6645725
🔍 Latest deploy log https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/683c13c7f92cb00008abc68c

@corsettigyg
Copy link
Copy Markdown
Collaborator

  • For reference, I created the disable_owner_inheritance to avoid making it into a breaking change, although I particularly believe that the default behaviour should be enable_owner_inheritance and this one should be set to False by default. To avoid headaches, I kept things as they are though.

  • Since the owners are being propagated in many places and now it is possible to disable/enable this behaviour, creating the tests is no pleasant task since it is a very repetitive process if we want to hit the coverage on the changed lines for all situations. I ended up using AI to automate most of the test creation since it would be very time consuming to do it manually. On a quick look the tests looked fine to me, but as a traditional data engineer I might still take a better look later once I have more time 😄

@CorsettiS CorsettiS marked this pull request as ready for review May 27, 2025 13:09
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. area:config Related to configuration, like YAML files, environment variables, or executer configuration area:docs Relating to documentation, changes, fixes, improvement labels May 27, 2025
Copy link
Copy Markdown
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, @CorsettiS , thanks a lot for contributing! Since it is a new feature, we'll release it as part of Cosmos 1.11.

I just enabled the integration tests and left a minor feedback inline.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 28, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.02%. Comparing base (003969e) to head (6645725).
Report is 4 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1787   +/-   ##
=======================================
  Coverage   98.02%   98.02%           
=======================================
  Files          85       85           
  Lines        5256     5263    +7     
=======================================
+ Hits         5152     5159    +7     
  Misses        104      104           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Contributor

@pankajkoti pankajkoti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. couple of minor suggestions in-line

Comment thread docs/configuration/render-config.rst Outdated
Comment thread cosmos/config.py Outdated
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Copy link
Copy Markdown
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @corsettigyg could we rename disable_owner_inheritance to enable_owner_inheritance to avoid double negation? I had commented this inline, but I think it got lost in one of the previous commits. We can set it to True to avoid being a breaking change, as you suggested - and we can review the default in Cosmos 2.x

@corsettigyg
Copy link
Copy Markdown
Collaborator

@tatiana done 😄 I thought about adding a deprecation warning to it but it would be annoying to have a constant warning showing up.

@linchun3
Copy link
Copy Markdown
Contributor

Thanks for this PR! It should help to close #1728

Any idea on the timeline for the release of Cosmos 1.11?

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Jun 3, 2025
@tatiana tatiana added this to the Cosmos 1.10.2 milestone Jun 3, 2025
@tatiana
Copy link
Copy Markdown
Collaborator

tatiana commented Jun 3, 2025

Thanks a lot, @corsettigyg ! @linchun3 We'll release this in 1.10.2 (although it can be perceived as a feature, it is also a sort of bug fix for some users). The estimate is to release it on 6 June (Friday)

@tatiana tatiana merged commit cfb814c into astronomer:main Jun 3, 2025
92 checks passed
@tatiana tatiana mentioned this pull request Aug 7, 2025
tatiana pushed a commit that referenced this pull request Aug 7, 2025
…1787)

From slack
`Hey everyone! 👋 I'm working on an Airflow DAG that uses
DbtTaskGroup (Cosmos v1.10.1). I've noticed that the Airflow tasks are
showing both the DAG owner (set in default_args) and the owner defined
in my dbt model's _doc_.yml. Is there a way to configure Cosmos to not
pull the owner from the dbt model? I'm aiming to keep the DAG owner
clean. Any insights would be greatly`

This PR introduces the flag `disable_owner_inheritance` which should
make it possible

(cherry picked from commit cfb814c)
tatiana added a commit that referenced this pull request Aug 8, 2025
**Bug Fixes**

* Fix task instance ``try_number`` attribute for Airflow 3 compatibility
by @pankajkoti in #1781
* Fix rendered template override logic when
``should_store_compiled_sql=False`` to restore pre-refactor behaviour by
@pankajkoti in #1777
* Fix ``ProfileConfig`` in GCP Cloud Run job execution mode by
@ramonvermeulen in #1783
* Fix dbt Docs page height by @1cadumagalhaes in #1793
* Add support to base64 encoded pem in Snowflake profiles by
@brunocmartins in #1801
* Allow to disable owner inheritance from dbt into airflow DAG owners by
@CorsettiS in #1787
* Fix Kubernetes Pod Operator conversion of ``container_resources`` to
``resources`` by @johnhoran in #1821
* Fix ``dbt deps`` with project level variables by @AlexandrKhabarov in
#1822
* Fix source freshness warnings in kubernetes execution mode by
@Pawel-Drabczyk in #1859
* Fix: Harden DbtNode against null config/meta by @pankajkoti in #1877
* Fix cache behaviour when DAG name contains "." by @tatiana in #1908

**Documentation**

* Fix ``contributing.rst`` docs by @tatiana in #1785
* Fix docs rendering in Airflow 3 Compatibility by @pankajastro in #1790
* Fix typo in ``selecting-excluding.rst`` by @msshroff in #1814
* Update testing behavior file with ``ExecutionMode.KUBERNETES`` by
@LuigiCerone in #1813
* Add step to fork repo in contributing guide by @pankajastro in #1808
* Fix ``depends_on`` attribute by @benedikt-buchert in #1837
* Fix character name by @ThePsyjo in #1860
* Update suggested MWAA startup script by @jaklan in #1884
* Make implementation & docs consistent regarding
``use_dataset_airflow3_uri_standard`` by @Anti0ff in #1878

**Others**

* Set retries to 0 in example DAGs by @pankajkoti in #1782
* Fix ``test_async_example_dag_without_setup_task`` tests by
@pankajastro in #1788
* Fix test hash value for Darwin when using Py 3.12.10 by @tatiana in
#1786
* Upgrade Python and Airflow used to run MyPy checks by @tatiana in
#1796
* Assert example DAGs' ``DagRunState`` and fix issues by @pankajkoti and
@tatiana in #1778
* Update the conflict matrix to include AF 2.10, 2.11 & 3.0 and dbt 1.9
& 1.10 by @tatiana in #1820
* Fix broken CI due to Pydantic conflicts by @tatiana in #1809
* Drop Python 3.8 Support by @pankajastro in #1852
* Add Airflow 2.11 to the test matrix by @tatiana in #1807
* Require Authorize for all jobs on pull requests from external
contributors in CI by @pankajkoti in #1861
* Leverage Trusted Publisher Management when publishing PyPI package by
@tatiana in #1862
* CI: Add back accidentally deleted python-version matrix for running
unit tests by @pankajkoti in #1872
* Remove commented code and fix mypy failures by @pankajkoti in #1876
* Add Zizmor analysis GitHub action by @pankajkoti in #1870
* Catch FlushError on Datasets for Airflow 2.11 dags test by @pankajkoti
in #1880
* Add deprecation warning for ``LoadMode.CUSTOM`` parser by
@duongphannamhung in #1885
* CI: Add GitHub CodeQL analysis workflow (codeql.yml) by @pankajkoti in
#1871
* Resolve 'credential persistence through GitHub Actions artifacts'
warnings from Zizmor by @pankajkoti in #1890
* Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti
in #1889
* Resolve Zizmor error alerts for unpinned image references; mark alert
for pull_request_target ignored by @pankajkoti in #1888
* Fix broken CI ``tests.py3.11-2.8-1.9:test-integration-setup`` by
@tatiana in #1902
* Add dbt-core 1.10 to test matrix by @tatiana in #1767
* Pin package dbt-databricks by @pankajastro in #1909
* Enable matrix test entry for dbt-1.9, python-3.9 and airflow-3.0 tests
in CI by @pankajastro in #1900
* Pre-commit updates: #1779, #1795, #1800, #1857, #1863, #1869, #1892,
#1901
* Dependabot updates: #1904

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
@pankajkoti pankajkoti mentioned this pull request Aug 8, 2025
tatiana added a commit that referenced this pull request Aug 8, 2025
**Bug Fixes**

* Fix task instance ``try_number`` attribute for Airflow 3 compatibility
by @pankajkoti in #1781
* Fix rendered template override logic when
``should_store_compiled_sql=False`` to restore pre-refactor behaviour by
@pankajkoti in #1777
* Fix ``ProfileConfig`` in GCP Cloud Run job execution mode by
@ramonvermeulen in #1783
* Fix dbt Docs page height by @1cadumagalhaes in #1793
* Add support to base64 encoded pem in Snowflake profiles by
@brunocmartins in #1801
* Allow to disable owner inheritance from dbt into airflow DAG owners by
@CorsettiS in #1787
* Fix Kubernetes Pod Operator conversion of ``container_resources`` to
``resources`` by @johnhoran in #1821
* Fix ``dbt deps`` with project level variables by @AlexandrKhabarov in
#1822
* Fix source freshness warnings in kubernetes execution mode by
@Pawel-Drabczyk in #1859
* Fix: Harden DbtNode against null config/meta by @pankajkoti in #1877
* Fix cache behaviour when DAG name contains "." by @tatiana in #1908

**Documentation**

* Fix ``contributing.rst`` docs by @tatiana in #1785
* Fix docs rendering in Airflow 3 Compatibility by @pankajastro in #1790
* Fix typo in ``selecting-excluding.rst`` by @msshroff in #1814
* Update testing behavior file with ``ExecutionMode.KUBERNETES`` by
@LuigiCerone in #1813
* Add step to fork repo in contributing guide by @pankajastro in #1808
* Fix ``depends_on`` attribute by @benedikt-buchert in #1837
* Fix character name by @ThePsyjo in #1860
* Update suggested MWAA startup script by @jaklan in #1884
* Make implementation & docs consistent regarding
``use_dataset_airflow3_uri_standard`` by @Anti0ff in #1878

**Others**

* Set retries to 0 in example DAGs by @pankajkoti in #1782
* Fix ``test_async_example_dag_without_setup_task`` tests by
@pankajastro in #1788
* Fix test hash value for Darwin when using Py 3.12.10 by @tatiana in
#1786
* Upgrade Python and Airflow used to run MyPy checks by @tatiana in
#1796
* Assert example DAGs' ``DagRunState`` and fix issues by @pankajkoti and
@tatiana in #1778
* Update the conflict matrix to include AF 2.10, 2.11 & 3.0 and dbt 1.9
& 1.10 by @tatiana in #1820
* Fix broken CI due to Pydantic conflicts by @tatiana in #1809
* Drop Python 3.8 Support by @pankajastro in #1852
* Add Airflow 2.11 to the test matrix by @tatiana in #1807
* Require Authorize for all jobs on pull requests from external
contributors in CI by @pankajkoti in #1861
* Leverage Trusted Publisher Management when publishing PyPI package by
@tatiana in #1862
* CI: Add back accidentally deleted python-version matrix for running
unit tests by @pankajkoti in #1872
* Remove commented code and fix mypy failures by @pankajkoti in #1876
* Add Zizmor analysis GitHub action by @pankajkoti in #1870
* Catch FlushError on Datasets for Airflow 2.11 dags test by @pankajkoti
in #1880
* Add deprecation warning for ``LoadMode.CUSTOM`` parser by
@duongphannamhung in #1885
* CI: Add GitHub CodeQL analysis workflow (codeql.yml) by @pankajkoti in
#1871
* Resolve 'credential persistence through GitHub Actions artifacts'
warnings from Zizmor by @pankajkoti in #1890
* Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti
in #1889
* Resolve Zizmor error alerts for unpinned image references; mark alert
for pull_request_target ignored by @pankajkoti in #1888
* Fix broken CI ``tests.py3.11-2.8-1.9:test-integration-setup`` by
@tatiana in #1902
* Add dbt-core 1.10 to test matrix by @tatiana in #1767
* Pin package dbt-databricks by @pankajastro in #1909
* Enable matrix test entry for dbt-1.9, python-3.9 and airflow-3.0 tests
in CI by @pankajastro in #1900
* Pre-commit updates: #1779, #1795, #1800, #1857, #1863, #1869, #1892,
#1901
* Dependabot updates: #1904

Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
tatiana added a commit that referenced this pull request Sep 18, 2025
[Giovanni Corsetti
Silva](https://www.linkedin.com/in/giovanni-corsetti/) (@corsettigyg
@CorsettiS @giovannicorsetti) is a Core Data Platform Data Engineer at
[Get Your Guide](https://www.getyourguide.com/). He's a Brazilian based
in Berlin (Germany) and has been an early adopter of Cosmos, using it
regularly throughout his career.

Not only has he been using Cosmos since the early stages, but he has
consistently improved Cosmos since April 2023:

<img width="494" height="249" alt="Screenshot 2025-09-17 at 16 39 49"
src="https://github.com/user-attachments/assets/ff9b0055-7df6-47a4-8efa-83a30d5aee99"
/>
] m. ,
<img width="492" height="252" alt="Screenshot 2025-09-17 at 16 40 11"
src="https://github.com/user-attachments/assets/8402d883-eff4-46aa-9ae7-afe35bd39a6e"
/>

<img width="499" height="256" alt="Screenshot 2025-09-17 at 16 39 23"
src="https://github.com/user-attachments/assets/c6b84245-aae1-46e6-99e3-552553619dd9"
/>

His contributions include new features, enhancements, bug fixes and also
improvements on the local dev setup, as it can see in some of the many
contributions he's done to the project so far:

* #1787
* #1761
* #1695 
* #1693 
* #1663
* #1571
* #1449
* #1099
* #1091
* #419
* #402
* #355 
* #354 
* #225 

Additionally, he has been interacting with users in the #airflow-dbt
Slack channel in a very collaborative and supportive way.

We want to promote him as a Cosmos committer and maintainer for all
these, recognising his constant efforts and achievements towards our
community. Thank you very much, @corsettigyg!

From this moment, you have write permissions on the repo:
<img width="884" height="93" alt="Screenshot 2025-09-18 at 14 34 07"
src="https://github.com/user-attachments/assets/abcc8c42-87cf-4b32-a7be-81127190d164"
/>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:config Related to configuration, like YAML files, environment variables, or executer configuration area:docs Relating to documentation, changes, fixes, improvement lgtm This PR has been approved by a maintainer size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants