Make `profiles_yml_path` optional for `ExecutionMode.DOCKER` and `KUBERNETES` by MrBones757 · Pull Request #681 · astronomer/astronomer-cosmos

MrBones757 · 2023-11-16T12:23:21Z

Description

This PR moves validation of the profiles_yml_path to later in the dag generation process such that additional context can be gathered from ExecutionConfig to avoid failing unnecessarily when the file does not exist.

Related Issue(s)

closes #680
closes #656

Breaking Change?

N/A

Checklist

I have made corresponding changes to the documentation (if required)
I have added tests that prove my fix is effective or that my feature works

…h profile_mapping and profiles_yml_filepath are defined

netlify · 2023-11-16T12:23:27Z

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name	Link
🔨 Latest commit	`0e6de62`
🔍 Latest deploy log	https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/6556bda281560600083ac92c

codecov · 2023-11-16T13:33:35Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (e23a445) 92.77% compared to head (0e6de62) 92.88%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #681      +/-   ##
==========================================
+ Coverage   92.77%   92.88%   +0.10%     
==========================================
  Files          55       55              
  Lines        2243     2248       +5     
==========================================
+ Hits         2081     2088       +7     
+ Misses        162      160       -2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

tatiana

This is looking great, @MrBones757 , one minor comment, and we're good to merge!

tatiana

Thanks, @MrBones757 , when using ExecutionMode.KUBERNETES and LoadMode.DBT_MANIFEST, we really should not enforce the profiles_yml_path to be set.

tatiana · 2023-11-23T16:51:35Z

@MrBones757 This was released in 1.2.5 🎉

@perttus

Features * Add ProfileMapping for Vertica by @perttus in #540 and #688 * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649 * Add support to select using (some) graph operators when using LoadMode.CUSTOM and LoadMode.DBT_MANIFEST by @tatiana in #728 * Add cosmos/propagate_logs Airflow config support for disabling log pr… by @agreenburg in #648 * Add operator_args full_refresh as a templated field by @joppevos in #623 * Expose environment variables and dbt variables in ProjectConfig by @jbandoro in #735 Enhancements * Make Pydantic an optional dependency by @pixie79 in #736 * Create a symbolic link to dbt_packages when dbt_deps is False when using LoadMode.DBT_LS by @DanMawdsleyBA in #730 * Support no profile_config for ExecutionMode.KUBERNETES and ExecutionMode.DOCKER by @MrBones757 and @tatiana in #681 and #731 * Add aws_session_token for Athena mapping by @benjamin-awd in #663 Others * Replace flake8 for Ruff by @joppevos in #743 * Reduce code complexity to 8 by @joppevos in #738 * Update conflict matrix between Airflow and dbt versions by @tatiana in #731 * Speed up integration tests by @jbandoro in #732

@perttus

Features * Add ProfileMapping for Vertica by @perttus in #540 and #688 * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649 * Add support to select using (some) graph operators when using LoadMode.CUSTOM and LoadMode.DBT_MANIFEST by @tatiana in #728 * Add cosmos/propagate_logs Airflow config support for disabling log pr… by @agreenburg in #648 * Add operator_args full_refresh as a templated field by @joppevos in #623 * Expose environment variables and dbt variables in ProjectConfig by @jbandoro in #735 Enhancements * Make Pydantic an optional dependency by @pixie79 in #736 * Create a symbolic link to dbt_packages when dbt_deps is False when using LoadMode.DBT_LS by @DanMawdsleyBA in #730 * Support no profile_config for ExecutionMode.KUBERNETES and ExecutionMode.DOCKER by @MrBones757 and @tatiana in #681 and #731 * Add aws_session_token for Athena mapping by @benjamin-awd in #663 Others * Replace flake8 for Ruff by @joppevos in #743 * Reduce code complexity to 8 by @joppevos in #738 * Update conflict matrix between Airflow and dbt versions by @tatiana in #731 * Speed up integration tests by @jbandoro in #732

@DanMawdsleyBA

**Features** * Add `ProfileMapping` for Snowflake encrypted private key path by @ivanstillfront in #608 * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649 * Add `DbtDocsGCSOperator` for uploading dbt docs to GCS by @jbandoro in #616 * Add support to select using (some) graph operators when using `LoadMode.CUSTOM` and `LoadMode.DBT_MANIFEST` by @tatiana in #728 * Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in #648 * Add operator_args ``full_refresh`` as a templated field by @joppevos in #623 * Expose environment variables and dbt variables in ``ProjectConfig`` by @jbandoro in #735 **Enhancements** * Make Pydantic an optional dependency by @pixie79 in #736 * Create a symbolic link to `dbt_packages` when `dbt_deps` is False when using `LoadMode.DBT_LS` by @DanMawdsleyBA in #730 * Support no `profile_config` for `ExecutionMode.KUBERNETES` and `ExecutionMode.DOCKER` by @MrBones757 and @tatiana in #681 and #731 * Add `aws_session_token` for Athena mapping by @benjamin-awd in #663 **Others** * Replace flake8 for Ruff by @joppevos in #743 * Reduce code complexity to 8 by @joppevos in #738 * Update conflict matrix between Airflow and dbt versions by @tatiana in #731 * Speed up integration tests by @jbandoro in #732

…ERNETES` (astronomer#681) This PR moves validation of the `profiles_yml_path` to later in the dag generation process such that additional context can be gathered from `ExecutionConfig` to avoid failing unnecessarily when the file does not exist. Closes: astronomer#680 Closes: astronomer#656

@binhnq94

Bug fixes * Fix running models that use alias while supporting dbt versions by @binhnq94 in astronomer#662 * Make profiles_yml_path optional for ExecutionMode.DOCKER and KUBERNETES by @MrBones757 in astronomer#681 * Prevent overriding dbt profile fields with profile args of type or method by @jbandoro in astronomer#702 * Fix LoadMode.DBT_LS fail when dbt outputs WarnErrorOptions by @adammarples in astronomer#692 * Add support for env vars in RenderConfig for dbt ls parsing by @jbandoro in astronomer#690 * Add support for Kubernetes on_warning_callback by @david-mag in astronomer#673 * Fix ExecutionConfig.dbt_executable_path to use ``default_factory`` by @jbandoro in astronomer#678 Others * Docs fix: example DAG in the README and docs/index by @tatiana in astronomer#705 * Docs improvement: highlight DAG examples in README by @iancmoritz and @jlaneve in astronomer#695 (cherry picked from commit 2878d6a)

@DanMawdsleyBA

**Features** * Add `ProfileMapping` for Snowflake encrypted private key path by @ivanstillfront in astronomer#608 * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in astronomer#649 * Add `DbtDocsGCSOperator` for uploading dbt docs to GCS by @jbandoro in astronomer#616 * Add support to select using (some) graph operators when using `LoadMode.CUSTOM` and `LoadMode.DBT_MANIFEST` by @tatiana in astronomer#728 * Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in astronomer#648 * Add operator_args ``full_refresh`` as a templated field by @joppevos in astronomer#623 * Expose environment variables and dbt variables in ``ProjectConfig`` by @jbandoro in astronomer#735 **Enhancements** * Make Pydantic an optional dependency by @pixie79 in astronomer#736 * Create a symbolic link to `dbt_packages` when `dbt_deps` is False when using `LoadMode.DBT_LS` by @DanMawdsleyBA in astronomer#730 * Support no `profile_config` for `ExecutionMode.KUBERNETES` and `ExecutionMode.DOCKER` by @MrBones757 and @tatiana in astronomer#681 and astronomer#731 * Add `aws_session_token` for Athena mapping by @benjamin-awd in astronomer#663 **Others** * Replace flake8 for Ruff by @joppevos in astronomer#743 * Reduce code complexity to 8 by @joppevos in astronomer#738 * Update conflict matrix between Airflow and dbt versions by @tatiana in astronomer#731 * Speed up integration tests by @jbandoro in astronomer#732

Delayed validation of profiles yml existance, added validation if bot…

3947238

…h profile_mapping and profiles_yml_filepath are defined

MrBones757 requested a review from a team as a code owner November 16, 2023 12:23

MrBones757 requested a review from a team November 16, 2023 12:23

MrBones757 temporarily deployed to external November 16, 2023 12:23 — with GitHub Actions Inactive

dosubot Bot added area:config Related to configuration, like YAML files, environment variables, or executer configuration execution:local Related to Local execution environment labels Nov 16, 2023

tatiana added this to the 1.2.5 milestone Nov 16, 2023

updated tests for coverage

dfdddd7

MrBones757 temporarily deployed to external November 16, 2023 14:40 — with GitHub Actions Inactive

tatiana reviewed Nov 16, 2023

View reviewed changes

Comment thread cosmos/converter.py Outdated

Merge branch 'main' into fix-profiles-yml-validation

7757bc9

tatiana had a problem deploying to external November 17, 2023 00:08 — with GitHub Actions Error

add additional validation to ensure value exists

0e6de62

MrBones757 temporarily deployed to external November 17, 2023 01:11 — with GitHub Actions Inactive

tatiana changed the title ~~enforce profiles_yml_path exists only for ExecutionMode.LOCAL and VIRTUALENV~~ Make profiles_yml_path optional for ExecutionMode.DOCKER and KUBERNETES Nov 18, 2023

tatiana approved these changes Nov 18, 2023

View reviewed changes

tatiana merged commit d28d3df into astronomer:main Nov 18, 2023

This was referenced Nov 23, 2023

DO NOT MERGE - Release 1.2.5 #708

Closed

Release 1.2.5 #709

Merged

Release 1.2.5 #710

Merged

tatiana mentioned this pull request Dec 7, 2023

Release 1.3.0a3 #753

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make `profiles_yml_path` optional for `ExecutionMode.DOCKER` and `KUBERNETES`#681

Make `profiles_yml_path` optional for `ExecutionMode.DOCKER` and `KUBERNETES`#681
tatiana merged 4 commits into
astronomer:mainfrom
MrBones757:fix-profiles-yml-validation

MrBones757 commented Nov 16, 2023

Uh oh!

netlify Bot commented Nov 16, 2023 •

edited

Loading

Uh oh!

codecov Bot commented Nov 16, 2023 •

edited

Loading

Uh oh!

tatiana left a comment

Uh oh!

Uh oh!

tatiana left a comment

Uh oh!

tatiana commented Nov 23, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

MrBones757 commented Nov 16, 2023

Description

Related Issue(s)

Breaking Change?

Checklist

Uh oh!

netlify Bot commented Nov 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Uh oh!

codecov Bot commented Nov 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

tatiana commented Nov 23, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

netlify Bot commented Nov 16, 2023 •

edited

Loading

codecov Bot commented Nov 16, 2023 •

edited

Loading