Add missing `DbtSourceGcpCloudRunJobOperator` in module `cosmos.operators.gcp_cloud_run_job` by anai-s · Pull Request #1290 · astronomer/astronomer-cosmos

anai-s · 2024-10-30T16:24:34Z

Description

this PR add a missing attribute DbtSourceGcpCloudRunJobOperator in cosmos.operators.gcp_cloud_run_job.py to be able to use the source_rendering_behavior params in cosmos.config.RenderConfig class with the new GCP_CLOUD_RUN_JOB execution mode (available in 1.7.0) .

Related Issue(s)

closes #1276

Breaking Change?

Checklist

I have made corresponding changes to the documentation (if required)
I have added tests that prove my fix is effective or that my feature works

… 'cosmos.operators.gcp_cloud_run_job'

netlify · 2024-10-30T16:24:59Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`c94928e`
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/672487aa816f080008bf7445

anai-s · 2024-10-31T08:51:15Z

Hello @tatiana My PR is ready for a review.

tatiana

@anai-s this looks great! Thanks for reporting the issue and creating the PR to fix it.

Could we also have a test related to this?

It may be possible to accomplish this by extending an existing test, such as test_dbt_gcp_cloud_run_job_build_command:
https://github.com/astronomer/astronomer-cosmos/blob/main/tests/operators/test_gcp_cloud_run_job.py

Tests are important so we make sure we don't have a regression on this in the future.

anai-s · 2024-10-31T16:28:02Z

@tatiana sure I will add the test! Thank you for the review.

anai-s · 2024-11-01T19:38:25Z

@tatiana I added the test for the module DbtSourceGcpCloudRunJobOperator. Can you review it please?

codecov · 2024-11-07T11:12:39Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.85%. Comparing base (1840c32) to head (311894b).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1290   +/-   ##
=======================================
  Coverage   95.85%   95.85%           
=======================================
  Files          67       67           
  Lines        3976     3979    +3     
=======================================
+ Hits         3811     3814    +3     
  Misses        165      165

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

tatiana

Looks great, thank you very much, @anai-s , for fixing this problem and adding the test. We'll release this as part of 1.8.0

@wornjs

**New Features** * Support customizing Airflow operator arguments per dbt node by @wornjs in #1339. [More information](https://astronomer.github.io/astronomer-cosmos/getting_started/custom-airflow-properties.html). * Support uploading dbt artifacts to remote cloud storages via callback by @pankajkoti in #1389. [Read more](https://astronomer.github.io/astronomer-cosmos/configuration/callbacks.html). * Add support to ``TestBehavior.BUILD`` by @tatiana in #1377. [Documentation](https://astronomer.github.io/astronomer-cosmos/configuration/testing-behavior.html). * Add support for the "at" operator when using ``LoadMode.DBT_MANIFEST`` or ``CUSTOM`` by @benjy44 in #1372 * Add dbt clone operator by @pankajastro in #1326, as documented in [here](https://astronomer.github.io/astronomer-cosmos/getting_started/operators.html). * Support rendering tasks with non-ASCII characters by @t0momi219 in #1278 [Read more](https://astronomer.github.io/astronomer-cosmos/configuration/task-display-name.html) * Add warning callback on source freshness by @pankajastro in #1400 [Read more](https://astronomer.github.io/astronomer-cosmos/configuration/source-nodes-rendering.html#on-warning-callback-callback) * Add Oracle Profile mapping by @slords and @pankajkoti in #1190 and #1404 * Emit telemetry to Scarf during DAG run by @tatiana in #1397 * Save tasks map as ``DbtToAirflowConverter`` property by @internetcoffeephone and @hheemskerk in #1362 **Bug Fixes** * Fix the mock value of port in ``TrinoBaseProfileMapping`` to be an integer by @dwolfeu #1322 * Fix access to the ``dbt docs`` menu item outside of Astro cloud by @tatiana in #1312 * Add missing ``DbtSourceGcpCloudRunJobOperator`` in module ``cosmos.operators.gcp_cloud_run_job`` by @anai-s in #1290 * Support building ``DbtDag`` without setting paths in ``ProjectConfig`` by @tatiana in #1307 * Fix parsing dbt ls outputs that contain JSONs that are not dbt nodes by @tatiana in #1296 * Fix Snowflake Profile mapping when using AWS default region by @tatiana in #1406 * Fix dag rendering for taskflow + DbtTaskGroup combo by @pankajastro in #1360 **Enhancements** * Improve dbt command execution logs to troubleshoot ``None`` values by @tatiana in #1392 * Add logging of stdout to dbt graph run_command by @KarolGongola in #1390 * Save tasks map as DbtToAirflowConverter property by @internetcoffeephone and @hheemskerk in #1362 * Support rendering build operator task-id with non-ASCII characters by @pankajastro in #1415 **Docs** * Remove extra ` char from docs by @pankajastro in #1345 * Add limitation about copying target dir files to remote by @pankajkoti in #1305 * Generalise example from README by @ReadytoRocc in #1311 * Add security policy by @tatiana, @chaosmaw and @lzdanski in # 1385 * Mention in documentation that the callback functionality is supported in ``ExecutionMode.VIRTUALENV`` by @pankajkoti in #1401 **Others** * Restore Jaffle Shop so that ``basic_cosmos_dag`` works as documented by @tatiana in #1374 * Remove Pytest durations from tests scripts by @tatiana in #1383 * Remove typing-extensions as dependency by @pankajastro in #1381 * Pin dbt-databricks version to < 1.9 by @pankajastro in #1376 * Refactor ``dbt-sqlite`` tests to use ``dbt-postgres`` by @pankajastro in #1366 * Remove 'dbt-core<1.8.9' pin by @tatiana in #1371 * Remove dependency ``eval_type_backport`` by @tatiana in #1370 * Enable kubernetes tests for dbt>=1.8 by @pankajastro #1364 * CI Workaround: Pin dbt-core, Disable SQLite Tests, and Correctly Ignore Clone Test to Pass CI by @pankajastro in #1337 * Enable Azure task in the remote store manifest example DAG by @pankajkoti in #1333 * Enable GCP remote manifest task by @pankajastro in #1332 * Add exempt label option in GH action stale job by @pankajastro in #1328 * Add integration test for source node rendering by @pankajastro in #1327 * Fix vulnerability issue on docs dependency by @tatiana in #1313 * Add postgres pod status check for k8s tests in CI by @pankajkoti in #1320 * [CI] Reduce the amount taking to run tests in the CI from 5h to 11min by @tatiana in #1297 * Enable secret detection precommit check by @pankajastro in #1302 * Fix security vulnerability, by not pinning Airflow 2.10.0 by @tatiana in #1298 * Fix Netlify build timeouts by @tatiana in #1294 * Add stalebot to label/close stale PRs and issues by @tatiana in #1288 * Unpin dbt-databricks version by @pankajastro in #1409 * Fix source resource type tests by @pankajastro in #1405 * Increase performance tests models by @tatiana in #1403 * Drop running 1000 models in the CI by @pankajkoti in #1411 * Fix releasing package to PyPI by @tatiana in #1396 * Pre-commit hook updates in #1394, #1373, #1358, #1340, #1331, #1314, #1301 Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Pankaj Singh <pankaj.singh@astronomer.io> Closes: #1193 --------- Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>

fix:add missing attribute 'DbtSourceGcpCloudRunJobOperator' in module…

5d5a254

… 'cosmos.operators.gcp_cloud_run_job'

dosubot Bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Oct 30, 2024

anai-s had a problem deploying to external October 30, 2024 16:24 — with GitHub Actions Error

dosubot Bot added the area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc label Oct 30, 2024

anai-s marked this pull request as draft October 30, 2024 16:28

Merge branch 'main' into fix-cloudrunjob-operator

b07a8d5

anai-s temporarily deployed to external October 30, 2024 16:29 — with GitHub Actions Inactive

anai-s marked this pull request as ready for review October 30, 2024 22:58

Merge branch 'main' into fix-cloudrunjob-operator

34b9889

anai-s temporarily deployed to external October 31, 2024 13:21 — with GitHub Actions Inactive

tatiana reviewed Oct 31, 2024

View reviewed changes

Merge branch 'main' into fix-cloudrunjob-operator

2248385

anai-s had a problem deploying to external October 31, 2024 16:12 — with GitHub Actions Error

add test for DbtSourceGcpCloudRunJobOperator class

5a8a351

anai-s had a problem deploying to external October 31, 2024 17:31 — with GitHub Actions Error

update test

b45b371

anai-s had a problem deploying to external November 1, 2024 07:30 — with GitHub Actions Error

add new condition to test

c94928e

anai-s had a problem deploying to external November 1, 2024 07:48 — with GitHub Actions Error

Merge branch 'main' into fix-cloudrunjob-operator

6eb8071

tatiana temporarily deployed to external November 7, 2024 09:18 — with GitHub Actions Inactive

tatiana changed the title ~~Add missing attribute 'DbtSourceGcpCloudRunJobOperator' in module 'cosmos.operators.gcp_cloud_run_job'~~ Add missing attribute DbtSourceGcpCloudRunJobOperator in module cosmos.operators.gcp_cloud_run_job Nov 11, 2024

tatiana changed the title ~~Add missing attribute DbtSourceGcpCloudRunJobOperator in module cosmos.operators.gcp_cloud_run_job~~ Add missing DbtSourceGcpCloudRunJobOperator in module cosmos.operators.gcp_cloud_run_job Nov 11, 2024

Merge branch 'main' into fix-cloudrunjob-operator

311894b

tatiana temporarily deployed to external November 11, 2024 15:22 — with GitHub Actions Inactive

tatiana added this to the Cosmos 1.8.0 milestone Nov 11, 2024

tatiana approved these changes Nov 11, 2024

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label Nov 11, 2024

tatiana merged commit a5de8b4 into astronomer:main Nov 11, 2024

tatiana mentioned this pull request Dec 17, 2024

Release 1.8.0 #1395

Merged

tatiana mentioned this pull request Feb 4, 2025

Add AWS ECS task run execution mode #1507

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add missing `DbtSourceGcpCloudRunJobOperator` in module `cosmos.operators.gcp_cloud_run_job`#1290

Add missing `DbtSourceGcpCloudRunJobOperator` in module `cosmos.operators.gcp_cloud_run_job`#1290
tatiana merged 9 commits into
astronomer:mainfrom
anai-s:fix-cloudrunjob-operator

anai-s commented Oct 30, 2024 •

edited

Loading

Uh oh!

netlify Bot commented Oct 30, 2024 •

edited

Loading

Uh oh!

anai-s commented Oct 31, 2024

Uh oh!

tatiana left a comment

Uh oh!

anai-s commented Oct 31, 2024

Uh oh!

anai-s commented Nov 1, 2024

Uh oh!

codecov Bot commented Nov 7, 2024 •

edited

Loading

Uh oh!

tatiana left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

anai-s commented Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Breaking Change?

Checklist

Uh oh!

netlify Bot commented Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Uh oh!

anai-s commented Oct 31, 2024

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

anai-s commented Oct 31, 2024

Uh oh!

anai-s commented Nov 1, 2024

Uh oh!

codecov Bot commented Nov 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

anai-s commented Oct 30, 2024 •

edited

Loading

netlify Bot commented Oct 30, 2024 •

edited

Loading

codecov Bot commented Nov 7, 2024 •

edited

Loading