Skip to content

Consistently handle build and imports in cosmos/__init__.py#1215

Merged
tatiana merged 1 commit into
mainfrom
add-build-operators-to-import
Sep 23, 2024
Merged

Consistently handle build and imports in cosmos/__init__.py#1215
tatiana merged 1 commit into
mainfrom
add-build-operators-to-import

Conversation

@tatiana
Copy link
Copy Markdown
Collaborator

@tatiana tatiana commented Sep 23, 2024

While reviewing #1153, I noticed we were inconsistent with Build operators, especially how we import and expose them in Cosmos __init__.py This PR addresses this.

These were the changes introduced:

  • Add DbtBuildAzureContainerInstanceOperator
  • Expose the build operators for all execution modes in Cosmos __init__ (missing for most, except ExecutionMode.LOCAL)
  • Be consistent with import error messages in __init__ (we were missing AWS EKS)

@tatiana
Add DbtBuildAzureContainerInstanceOperator and expose all operators i…
3ca4e0a 
…n Cosmos __init__, including missing EKS and a few Build ones
@netlify
Copy link
Copy Markdown

netlify Bot commented Sep 23, 2024 •
edited
Loading

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit 3ca4e0a
🔍 Latest deploy log https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/66f112632916f1000811d765

@tatiana tatiana marked this pull request as ready for review September 23, 2024 07:49
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc dbt:build Primarily related to dbt build command or functionality labels Sep 23, 2024
@tatiana tatiana mentioned this pull request Sep 23, 2024
Merged
2 tasks
pankajastro
pankajastro approved these changes Sep 23, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@pankajastro pankajastro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Sep 23, 2024
@codecov
Copy link
Copy Markdown

codecov Bot commented Sep 23, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 43.75000% with 9 lines in your changes missing coverage. Please review.

Project coverage is 95.72%. Comparing base (11de5ba) to head (3ca4e0a).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
cosmos/__init__.py 25.00% 9 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #1215      +/-   ##
==========================================
- Coverage   95.95%   95.72%   -0.23%     
==========================================
  Files          64       64              
  Lines        3656     3672      +16     
==========================================
+ Hits         3508     3515       +7     
- Misses        148      157       +9
Flag Coverage Δ
95.72% <43.75%> (-0.23%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pankajastro
Copy link
Copy Markdown
Contributor

pankajastro commented Sep 23, 2024

Not in this PR but @tatiana do you think it makes sense to add some script in pre-commit to check the class from the constant and operator module included in __init__ file?

@pankajastro
Copy link
Copy Markdown
Contributor

pankajastro commented Sep 23, 2024

There was some similar fix #1184 in the past also but looks like we missed some

@tatiana tatiana mentioned this pull request Sep 23, 2024
Closed
@tatiana
Copy link
Copy Markdown
Collaborator Author

tatiana commented Sep 23, 2024

Thanks for the light-of-speed review, @pankajastro !

It's not in this PR, but @tatiana, do you think it makes sense to add a script in the pre-commit to check the class from the constant and operator module included in the init file?
Yes, I believe that it makes sense to automate the check for imports, I logged a ticket so we don't forget this: #1216

In the mid-long term, I'd love to stop this practice of having everything exposed in cosmos/__init__.py. People could just import them from other locations. I logged a ticket for this: #1213

@tatiana tatiana merged commit 0fd0bdd into main Sep 23, 2024
@tatiana tatiana deleted the add-build-operators-to-import branch September 23, 2024 08:29
@tatiana tatiana added this to the Cosmos 1.7.0 milestone Sep 23, 2024
@tatiana tatiana mentioned this pull request Oct 2, 2024
Merged
tatiana added a commit that referenced this pull request Oct 4, 2024
@tatiana
Release 1.7.0
cd8a4f5 
New Features

* Introduction of experimental support to run dbt BQ models using Airflow deferrable operators by @pankajkoti @pankajastro @tatiana in #1224 #1230.
  This is a first step in this journey and we would really appreciate feedback from the community.

  For more information, check the documentation: https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes.html#airflow-async-experimental

  This work has been inspired by the talk "Airflow at Monzo: Evolving our data platform as the bank scales" by
  @jonathanrainer @ed-sparkes given at Airflow Summit 2023: https://airflowsummit.org/sessions/2023/airflow-at-monzo-evolving-our-data-platform-as-the-bank-scales/.

* Support using ``DatasetAlias`` and fix orphaning unreferenced dataset by @tatiana in #1217 #1240

  Documentation: https://astronomer.github.io/astronomer-cosmos/configuration/scheduling.html#data-aware-scheduling

* Add GCP_CLOUD_RUN_JOB execution mode by @ags-de #1153

  Learn more about it: https://astronomer.github.io/astronomer-cosmos/getting_started/gcp-cloud-run-job.html

Enhancements

* Create single virtualenv when ``DbtVirtualenvBaseOperator`` has ``virtualenv_dir=None`` and ``is_virtualenv_dir_temporary=True`` by @kesompochy in #1200
* Consistently handle build and imports in ``cosmos/__init__.py`` by @tatiana in #1215
* Add enum constants to init for direct import by @fabiomx in #1184

Bug fixes

* URL encode dataset names to support multibyte characters by @t0momi219 in #1198
* Fix invalid argument (``full_refresh``) passed to DbtTestAwsEksOperator (and others) by @johnhoran in #1175
* Fix ``printer_width`` arg type in ``DbtProfileConfigVars`` by @jessicaschueler in #1191
* Fix task owner fallback by @jmaicher in #1195

Docs

* Add scarf to readme and docs for website analytics by @cmarteepants in #1221
* Add ``virtualenv_dir`` param to ``ExecutionConfig`` docs by @pankajkoti in #1173
* Give credits to @LennartKloppenburg in CHANGELOG.rst by @tatiana #1174
* Refactor docs for async mode execution by @pankajkoti in #1241

Others

* Remove PR branch added for testing a change in CI in #1224 by @pankajkoti in #1233
* Fix CI wrt broken coverage upload artifact @pankajkoti in #1210
* Fix CI issues - Upgrade actions/upload-artifact & actions/download-artifact to v4 and set min version for packaging by @pankajkoti in #1208
* Resolve CI failures for Apache Airflow 2.7 jobs by @pankajkoti in #1182
* CI: Update GCP manifest file path based on new secret update by @pankajkoti in #1237
* Pre-commit hook updates in #1176 #1186, #1186, #1201, #1219, #1231
tatiana added a commit that referenced this pull request Oct 4, 2024
@tatiana
Release 1.7.0
fd01b1d 
New Features

* Introduction of experimental support to run dbt BQ models using Airflow deferrable operators by @pankajkoti @pankajastro @tatiana in #1224 #1230.
  This is a first step in this journey and we would really appreciate feedback from the community.

  For more information, check the documentation: https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes.html#airflow-async-experimental

  This work has been inspired by the talk "Airflow at Monzo: Evolving our data platform as the bank scales" by
  @jonathanrainer @ed-sparkes given at Airflow Summit 2023: https://airflowsummit.org/sessions/2023/airflow-at-monzo-evolving-our-data-platform-as-the-bank-scales/.

* Support using ``DatasetAlias`` and fix orphaning unreferenced dataset by @tatiana in #1217 #1240

  Documentation: https://astronomer.github.io/astronomer-cosmos/configuration/scheduling.html#data-aware-scheduling

* Add GCP_CLOUD_RUN_JOB execution mode by @ags-de #1153

  Learn more about it: https://astronomer.github.io/astronomer-cosmos/getting_started/gcp-cloud-run-job.html

Enhancements

* Create single virtualenv when ``DbtVirtualenvBaseOperator`` has ``virtualenv_dir=None`` and ``is_virtualenv_dir_temporary=True`` by @kesompochy in #1200
* Consistently handle build and imports in ``cosmos/__init__.py`` by @tatiana in #1215
* Add enum constants to init for direct import by @fabiomx in #1184

Bug fixes

* URL encode dataset names to support multibyte characters by @t0momi219 in #1198
* Fix invalid argument (``full_refresh``) passed to DbtTestAwsEksOperator (and others) by @johnhoran in #1175
* Fix ``printer_width`` arg type in ``DbtProfileConfigVars`` by @jessicaschueler in #1191
* Fix task owner fallback by @jmaicher in #1195

Docs

* Add scarf to readme and docs for website analytics by @cmarteepants in #1221
* Add ``virtualenv_dir`` param to ``ExecutionConfig`` docs by @pankajkoti in #1173
* Give credits to @LennartKloppenburg in CHANGELOG.rst by @tatiana #1174
* Refactor docs for async mode execution by @pankajkoti in #1241

Others

* Remove PR branch added for testing a change in CI in #1224 by @pankajkoti in #1233
* Fix CI wrt broken coverage upload artifact @pankajkoti in #1210
* Fix CI issues - Upgrade actions/upload-artifact & actions/download-artifact to v4 and set min version for packaging by @pankajkoti in #1208
* Resolve CI failures for Apache Airflow 2.7 jobs by @pankajkoti in #1182
* CI: Update GCP manifest file path based on new secret update by @pankajkoti in #1237
* Pre-commit hook updates in #1176 #1186, #1186, #1201, #1219, #1231
tatiana added a commit that referenced this pull request Oct 4, 2024
@tatiana @pankajkoti
Release 1.7 (#1234)
534a14e 
**New Features**

* Support using ``DatasetAlias`` and fix orphaning unreferenced dataset
by @tatiana in #1217 #1240

Documentation:
https://astronomer.github.io/astronomer-cosmos/configuration/scheduling.html#data-aware-scheduling

* Add GCP_CLOUD_RUN_JOB execution mode by @ags-de #1153

Learn more about it:
https://astronomer.github.io/astronomer-cosmos/getting_started/gcp-cloud-run-job.html

* Introduction of experimental support to run dbt BQ models using
Airflow deferrable operators by @pankajkoti @pankajastro @tatiana in
#1224 #1230.

This is the first step in the journey of running dbt resources with
native Airflow, and we would appreciate feedback from the community.

For more information, check the documentation:
https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes.html#airflow-async-experimental

This work has been inspired by the talk "Airflow at Monzo: Evolving our
data platform as the bank scales" by
@jonathanrainer @ed-sparkes given at Airflow Summit 2023:
https://airflowsummit.org/sessions/2023/airflow-at-monzo-evolving-our-data-platform-as-the-bank-scales/.


**Enhancements**

* Create single virtualenv when ``DbtVirtualenvBaseOperator`` has
``virtualenv_dir=None`` and ``is_virtualenv_dir_temporary=True`` by
@kesompochy in #1200
* Consistently handle build and imports in ``cosmos/__init__.py`` by
@tatiana in #1215
* Add enum constants to init for direct import by @fabiomx in #1184

**Bug fixes**

* URL encode dataset names to support multibyte characters by @t0momi219
in #1198
* Fix invalid argument (``full_refresh``) passed to
DbtTestAwsEksOperator (and others) by @johnhoran in #1175
* Fix ``printer_width`` arg type in ``DbtProfileConfigVars`` by
@jessicaschueler in #1191
* Fix task owner fallback by @jmaicher in #1195

**Docs**

* Add scarf to readme and docs for website analytics by @cmarteepants in
#1221
* Add ``virtualenv_dir`` param to ``ExecutionConfig`` docs by
@pankajkoti in #1173
* Give credits to @LennartKloppenburg in CHANGELOG.rst by @tatiana #1174
* Refactor docs for async mode execution by @pankajkoti in #1241

Others

* Remove PR branch added for testing a change in CI in #1224 by
@pankajkoti in #1233
* Fix CI wrt broken coverage upload artifact @pankajkoti in #1210
* Fix CI issues - Upgrade actions/upload-artifact &
actions/download-artifact to v4 and set min version for packaging by
@pankajkoti in #1208
* Resolve CI failures for Apache Airflow 2.7 jobs by @pankajkoti in
#1182
* CI: Update GCP manifest file path based on new secret update by
@pankajkoti in #1237
* Pre-commit hook updates in #1176 #1186, #1186, #1201, #1219, #1231

---------

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pankajkoti pankajkoti pankajkoti approved these changes

@pankajastro pankajastro pankajastro approved these changes

Assignees

No one assigned

Labels

area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc dbt:build Primarily related to dbt build command or functionality lgtm This PR has been approved by a maintainer size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Milestone

Cosmos 1.7.0

Development

Successfully merging this pull request may close these issues.

3 participants

@tatiana @pankajastro @pankajkoti