Add GCP_CLOUD_RUN_JOB execution mode by ags-de · Pull Request #1153 · astronomer/astronomer-cosmos

ags-de · 2024-08-12T09:37:00Z

Description

Added new GCP_CLOUD_RUN_JOB execution mode that triggers Google Cloud Platform's Cloud Run Job instance with dbt model in it.

It extends Airflow's CloudRunExecuteJobOperator and overrides Cloud Run Job's container with dbt command generated by cosmos.

Note: CloudRunExecuteJobOperator has container_overrides parameter implemented in apache-airflow-providers-google==10.13.0 which is supported by airflow >=2.6.0.

Related Issue(s)

resolves #1149

Breaking Change?

None, the new execution mode is optional.

Checklist

I have made corresponding changes to the documentation (if required)
I have added tests that prove my fix is effective or that my feature works

netlify · 2024-08-12T09:37:28Z

✅ Deploy Preview for sunny-pastelito-5ecb04 ready!

Name	Link
🔨 Latest commit	`83f9a79`
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/66f5046ee1c738000877680e
😎 Deploy Preview	https://deploy-preview-1153--sunny-pastelito-5ecb04.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

pankajastro · 2024-08-21T10:34:40Z

Hi @ags-de, Thank you so much for your contribution. Please let us know if you need any support.

pankajkoti · 2024-08-29T16:08:28Z

hi @ags-de could you please rebase the PR. We just merged #1182 that should fix the breaking CI jobs.

codecov · 2024-08-31T19:08:34Z

Codecov Report

Attention: Patch coverage is 98.63014% with 1 line in your changes missing coverage. Please review.

Project coverage is 95.77%. Comparing base (3414513) to head (8e549ca).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
cosmos/operators/gcp_cloud_run_job.py	98.36%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1153      +/-   ##
==========================================
+ Coverage   95.72%   95.77%   +0.05%     
==========================================
  Files          64       65       +1     
  Lines        3672     3744      +72     
==========================================
+ Hits         3515     3586      +71     
- Misses        157      158       +1

Flag	Coverage Δ
	`95.77% <98.63%> (+0.05%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ags-de · 2024-09-25T12:49:10Z

Hello @tatiana,

The branch is rebased and I also fixed a few typos in docs. It should be ready to go.

tatiana

Hi @ags-de amazing work!
I left one last comment. I'll wait for your feedback/change before we merge into the main branch.

Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>

Added new `GCP_CLOUD_RUN_JOB` execution mode that triggers Google Cloud Platform's Cloud Run Job instance with dbt model in it. It extends Airflow's `CloudRunExecuteJobOperator` and overrides Cloud Run Job's container with dbt command generated by cosmos. Note: `CloudRunExecuteJobOperator` has `container_overrides` parameter implemented in `apache-airflow-providers-google==10.13.0` which is supported by `airflow >=2.6.0`. Resolves astronomer#1149 Co-authored-by: Agata Zalewska <azalewska5@grupazpr.pl> Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>

@pankajkoti

New Features * Introduction of experimental support to run dbt BQ models using Airflow deferrable operators by @pankajkoti @pankajastro @tatiana in #1224 #1230. This is a first step in this journey and we would really appreciate feedback from the community. For more information, check the documentation: https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes.html#airflow-async-experimental This work has been inspired by the talk "Airflow at Monzo: Evolving our data platform as the bank scales" by @jonathanrainer @ed-sparkes given at Airflow Summit 2023: https://airflowsummit.org/sessions/2023/airflow-at-monzo-evolving-our-data-platform-as-the-bank-scales/. * Support using ``DatasetAlias`` and fix orphaning unreferenced dataset by @tatiana in #1217 #1240 Documentation: https://astronomer.github.io/astronomer-cosmos/configuration/scheduling.html#data-aware-scheduling * Add GCP_CLOUD_RUN_JOB execution mode by @ags-de #1153 Learn more about it: https://astronomer.github.io/astronomer-cosmos/getting_started/gcp-cloud-run-job.html Enhancements * Create single virtualenv when ``DbtVirtualenvBaseOperator`` has ``virtualenv_dir=None`` and ``is_virtualenv_dir_temporary=True`` by @kesompochy in #1200 * Consistently handle build and imports in ``cosmos/__init__.py`` by @tatiana in #1215 * Add enum constants to init for direct import by @fabiomx in #1184 Bug fixes * URL encode dataset names to support multibyte characters by @t0momi219 in #1198 * Fix invalid argument (``full_refresh``) passed to DbtTestAwsEksOperator (and others) by @johnhoran in #1175 * Fix ``printer_width`` arg type in ``DbtProfileConfigVars`` by @jessicaschueler in #1191 * Fix task owner fallback by @jmaicher in #1195 Docs * Add scarf to readme and docs for website analytics by @cmarteepants in #1221 * Add ``virtualenv_dir`` param to ``ExecutionConfig`` docs by @pankajkoti in #1173 * Give credits to @LennartKloppenburg in CHANGELOG.rst by @tatiana #1174 * Refactor docs for async mode execution by @pankajkoti in #1241 Others * Remove PR branch added for testing a change in CI in #1224 by @pankajkoti in #1233 * Fix CI wrt broken coverage upload artifact @pankajkoti in #1210 * Fix CI issues - Upgrade actions/upload-artifact & actions/download-artifact to v4 and set min version for packaging by @pankajkoti in #1208 * Resolve CI failures for Apache Airflow 2.7 jobs by @pankajkoti in #1182 * CI: Update GCP manifest file path based on new secret update by @pankajkoti in #1237 * Pre-commit hook updates in #1176 #1186, #1186, #1201, #1219, #1231

@pankajkoti

New Features * Introduction of experimental support to run dbt BQ models using Airflow deferrable operators by @pankajkoti @pankajastro @tatiana in #1224 #1230. This is a first step in this journey and we would really appreciate feedback from the community. For more information, check the documentation: https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes.html#airflow-async-experimental This work has been inspired by the talk "Airflow at Monzo: Evolving our data platform as the bank scales" by @jonathanrainer @ed-sparkes given at Airflow Summit 2023: https://airflowsummit.org/sessions/2023/airflow-at-monzo-evolving-our-data-platform-as-the-bank-scales/. * Support using ``DatasetAlias`` and fix orphaning unreferenced dataset by @tatiana in #1217 #1240 Documentation: https://astronomer.github.io/astronomer-cosmos/configuration/scheduling.html#data-aware-scheduling * Add GCP_CLOUD_RUN_JOB execution mode by @ags-de #1153 Learn more about it: https://astronomer.github.io/astronomer-cosmos/getting_started/gcp-cloud-run-job.html Enhancements * Create single virtualenv when ``DbtVirtualenvBaseOperator`` has ``virtualenv_dir=None`` and ``is_virtualenv_dir_temporary=True`` by @kesompochy in #1200 * Consistently handle build and imports in ``cosmos/__init__.py`` by @tatiana in #1215 * Add enum constants to init for direct import by @fabiomx in #1184 Bug fixes * URL encode dataset names to support multibyte characters by @t0momi219 in #1198 * Fix invalid argument (``full_refresh``) passed to DbtTestAwsEksOperator (and others) by @johnhoran in #1175 * Fix ``printer_width`` arg type in ``DbtProfileConfigVars`` by @jessicaschueler in #1191 * Fix task owner fallback by @jmaicher in #1195 Docs * Add scarf to readme and docs for website analytics by @cmarteepants in #1221 * Add ``virtualenv_dir`` param to ``ExecutionConfig`` docs by @pankajkoti in #1173 * Give credits to @LennartKloppenburg in CHANGELOG.rst by @tatiana #1174 * Refactor docs for async mode execution by @pankajkoti in #1241 Others * Remove PR branch added for testing a change in CI in #1224 by @pankajkoti in #1233 * Fix CI wrt broken coverage upload artifact @pankajkoti in #1210 * Fix CI issues - Upgrade actions/upload-artifact & actions/download-artifact to v4 and set min version for packaging by @pankajkoti in #1208 * Resolve CI failures for Apache Airflow 2.7 jobs by @pankajkoti in #1182 * CI: Update GCP manifest file path based on new secret update by @pankajkoti in #1237 * Pre-commit hook updates in #1176 #1186, #1186, #1201, #1219, #1231

@tatiana

**New Features** * Support using ``DatasetAlias`` and fix orphaning unreferenced dataset by @tatiana in #1217 #1240 Documentation: https://astronomer.github.io/astronomer-cosmos/configuration/scheduling.html#data-aware-scheduling * Add GCP_CLOUD_RUN_JOB execution mode by @ags-de #1153 Learn more about it: https://astronomer.github.io/astronomer-cosmos/getting_started/gcp-cloud-run-job.html * Introduction of experimental support to run dbt BQ models using Airflow deferrable operators by @pankajkoti @pankajastro @tatiana in #1224 #1230. This is the first step in the journey of running dbt resources with native Airflow, and we would appreciate feedback from the community. For more information, check the documentation: https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes.html#airflow-async-experimental This work has been inspired by the talk "Airflow at Monzo: Evolving our data platform as the bank scales" by @jonathanrainer @ed-sparkes given at Airflow Summit 2023: https://airflowsummit.org/sessions/2023/airflow-at-monzo-evolving-our-data-platform-as-the-bank-scales/. **Enhancements** * Create single virtualenv when ``DbtVirtualenvBaseOperator`` has ``virtualenv_dir=None`` and ``is_virtualenv_dir_temporary=True`` by @kesompochy in #1200 * Consistently handle build and imports in ``cosmos/__init__.py`` by @tatiana in #1215 * Add enum constants to init for direct import by @fabiomx in #1184 **Bug fixes** * URL encode dataset names to support multibyte characters by @t0momi219 in #1198 * Fix invalid argument (``full_refresh``) passed to DbtTestAwsEksOperator (and others) by @johnhoran in #1175 * Fix ``printer_width`` arg type in ``DbtProfileConfigVars`` by @jessicaschueler in #1191 * Fix task owner fallback by @jmaicher in #1195 **Docs** * Add scarf to readme and docs for website analytics by @cmarteepants in #1221 * Add ``virtualenv_dir`` param to ``ExecutionConfig`` docs by @pankajkoti in #1173 * Give credits to @LennartKloppenburg in CHANGELOG.rst by @tatiana #1174 * Refactor docs for async mode execution by @pankajkoti in #1241 Others * Remove PR branch added for testing a change in CI in #1224 by @pankajkoti in #1233 * Fix CI wrt broken coverage upload artifact @pankajkoti in #1210 * Fix CI issues - Upgrade actions/upload-artifact & actions/download-artifact to v4 and set min version for packaging by @pankajkoti in #1208 * Resolve CI failures for Apache Airflow 2.7 jobs by @pankajkoti in #1182 * CI: Update GCP manifest file path based on new secret update by @pankajkoti in #1237 * Pre-commit hook updates in #1176 #1186, #1186, #1201, #1219, #1231 --------- Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

ags-de had a problem deploying to external August 12, 2024 09:37 — with GitHub Actions Error

pre-commit-ci Bot had a problem deploying to external August 12, 2024 09:38 Error

ags-de had a problem deploying to external August 13, 2024 06:41 — with GitHub Actions Error

pre-commit-ci Bot had a problem deploying to external August 13, 2024 06:41 Error

ags-de had a problem deploying to external August 14, 2024 07:54 — with GitHub Actions Error

pre-commit-ci Bot had a problem deploying to external August 14, 2024 07:54 Error

tatiana added this to the Cosmos 1.7.0 milestone Aug 14, 2024

ags-de had a problem deploying to external August 19, 2024 06:33 — with GitHub Actions Error

pre-commit-ci Bot had a problem deploying to external August 19, 2024 06:43 Error

ags-de changed the title ~~[WIP] Add GCP_CLOUD_RUN_JOB execution mode~~ Add GCP_CLOUD_RUN_JOB execution mode Aug 28, 2024

ags-de had a problem deploying to external August 28, 2024 09:15 — with GitHub Actions Error

pre-commit-ci Bot had a problem deploying to external August 28, 2024 09:15 Error

ags-de marked this pull request as ready for review August 28, 2024 09:18

dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc labels Aug 28, 2024

ags-de temporarily deployed to external August 29, 2024 16:33 — with GitHub Actions Inactive

ags-de had a problem deploying to external August 31, 2024 12:43 — with GitHub Actions Error

ags-de had a problem deploying to external August 31, 2024 14:26 — with GitHub Actions Error

pre-commit-ci Bot had a problem deploying to external August 31, 2024 14:27 Error

ags-de temporarily deployed to external August 31, 2024 16:30 — with GitHub Actions Inactive

ags-de had a problem deploying to external September 1, 2024 16:13 — with GitHub Actions Error

pre-commit-ci Bot had a problem deploying to external September 1, 2024 16:13 Error

ags-de force-pushed the gcpcloudrun-executor branch from 855dba3 to e30de3b Compare September 1, 2024 16:20

pre-commit-ci Bot had a problem deploying to external September 1, 2024 16:21 Error

tatiana modified the milestones: Cosmos 1.7.0, Triage Sep 20, 2024

ags-de had a problem deploying to external September 24, 2024 20:47 — with GitHub Actions Error

ags-de force-pushed the gcpcloudrun-executor branch from ab2f009 to 5dcec92 Compare September 24, 2024 21:09

ags-de had a problem deploying to external September 24, 2024 21:09 — with GitHub Actions Error

ags-de had a problem deploying to external September 24, 2024 21:13 — with GitHub Actions Error

ags-de had a problem deploying to external September 24, 2024 21:23 — with GitHub Actions Error

pre-commit-ci Bot had a problem deploying to external September 24, 2024 21:24 Error

ags-de force-pushed the gcpcloudrun-executor branch from 342a7bf to 0bb09a1 Compare September 24, 2024 22:11

pre-commit-ci Bot temporarily deployed to external September 24, 2024 22:11 Inactive

tatiana mentioned this pull request Sep 25, 2024

Review and unify many (public) Astronomer repos related to Cosmos #1222

Open

3 tasks

ags-de force-pushed the gcpcloudrun-executor branch from 51b8a24 to e55bc97 Compare September 25, 2024 07:38

ags-de had a problem deploying to external September 25, 2024 07:38 — with GitHub Actions Error

add gcp_cloud_run_job execution mode (astronomer#1149)

8e549ca

ags-de force-pushed the gcpcloudrun-executor branch from e55bc97 to 8e549ca Compare September 25, 2024 10:55

ags-de temporarily deployed to external September 25, 2024 10:55 — with GitHub Actions Inactive

tatiana reviewed Sep 25, 2024

View reviewed changes

Comment thread pyproject.toml Outdated

tatiana reviewed Sep 25, 2024

View reviewed changes

Comment thread pyproject.toml Outdated

tatiana approved these changes Sep 25, 2024

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label Sep 25, 2024

Update pyproject.toml

bb57926

Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>

ags-de temporarily deployed to external September 25, 2024 21:02 — with GitHub Actions Inactive

Merge branch 'main' into gcpcloudrun-executor

83f9a79

tatiana merged commit 56ff6dd into astronomer:main Sep 26, 2024

tatiana had a problem deploying to external September 26, 2024 06:51 — with GitHub Actions Failure

ags-de deleted the gcpcloudrun-executor branch September 26, 2024 10:28

tatiana mentioned this pull request Oct 2, 2024

Release 1.7 #1234

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GCP_CLOUD_RUN_JOB execution mode#1153

Add GCP_CLOUD_RUN_JOB execution mode#1153
tatiana merged 3 commits into
astronomer:mainfrom
ags-de:gcpcloudrun-executor

ags-de commented Aug 12, 2024 •

edited

Loading

Uh oh!

netlify Bot commented Aug 12, 2024 •

edited

Loading

Uh oh!

pankajastro commented Aug 21, 2024

Uh oh!

pankajkoti commented Aug 29, 2024

Uh oh!

codecov Bot commented Aug 31, 2024 •

edited

Loading

Uh oh!

ags-de commented Sep 25, 2024

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ags-de commented Aug 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Breaking Change?

Checklist

Uh oh!

netlify Bot commented Aug 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for sunny-pastelito-5ecb04 ready!

Uh oh!

pankajastro commented Aug 21, 2024

Uh oh!

pankajkoti commented Aug 29, 2024

Uh oh!

codecov Bot commented Aug 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ags-de commented Sep 25, 2024

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ags-de commented Aug 12, 2024 •

edited

Loading

netlify Bot commented Aug 12, 2024 •

edited

Loading

codecov Bot commented Aug 31, 2024 •

edited

Loading